Automated Data Labeling

Automating or semi-automating data labeling is feasible for big datasets including recognized items. The dataset will automatically be labeled using custom machine learning models that have been trained to identify particular sorts of data.

Only after creating early, high-quality ground-truth datasets can you make use of automatic data labeling. It can be difficult to take into consideration all edge circumstances and to fully trust automated data labeling to produce the best quality labels, even given high-quality ground truth.

Human Only Labeling

For many modalities we care about for machine learning applications, like vision and natural language processing, humans are extraordinarily excellent at certain tasks. In many domains, human data labeling produces labels of superior quality than automated data labeling.

However, it might be difficult to train people to consistently categorize the same data because human experiences can differ in their subjectivity. Moreover, the cost of human labeling for a given work can be higher than that of automated labeling due to their relative slowerness.

Human in the Loop (HITL) Labeling

Human-in-the-loop labeling makes use of people's extremely specialized skills to support automated data labeling. HITL data labeling can originate from actively tooled data that enhances the quality and efficiency of labeling, or from automatically labeled data audited by humans. The accuracy and efficiency of automated labeling with a human in the loop almost usually surpass those of either approach alone.

You will need to figure out how to source your labeling workforce if you decide to include humans in your data labeling workforce, which is something we strongly advise. Will you grow to a third-party labeling company, recruit a team internally, or persuade friends and relatives to label your data for free? Below, we offer a framework to assist you in making this choice.

In-House Teams

Small businesses might not have the funding to dedicate a large amount of resources to data labeling; as a result, all team members—including the CEO—may have to label data themselves. This method might work for a small prototype, but it is not scalable.

Large, well-funded companies may decide to maintain internal labeling teams in order to maintain control over the whole data flow. Although this method offers a great deal of control and flexibility, it is costly and requires a lot of management.

Businesses that handle sensitive data or have privacy concerns may decide to deploy internal labeling teams. Although a totally reasonable strategy, scaling this can be challenging.

Pros: Strict control over data pipelines and subject matter knowledge

Cons: High costs, associated with supervising and training labelers

Crowdsourcing

Crowdsourcing systems offer a rapid and simple method for a huge pool of people to swiftly execute a variety of activities. These platforms work well for labeling data that doesn't raise privacy issues, including publicly available datasets with straightforward instructions and annotations. However, the unskilled resource pool from crowdsource sites is a bad alternative if more complicated labels are required or if sensitive data is involved. Crowdsourcing platforms often contain resources that are poorly trained and lack domain expertise, which results in labeling that is of low quality.

Pros: Availability of a bigger pool of labelers

Cons: Questionable quality; high training and administrative costs for labelers

3rd Party Data Labeling Partners

Third-party data labeling organizations are proficient in producing high-quality data labels and frequently possess extensive knowledge of machine learning. These businesses can serve as your technical partners, offering guidance on the best ways to gather, organize, and classify your data as well as best practices for the full machine learning lifecycle. These businesses provide high-quality labels at competitive prices thanks to their highly skilled resource pools, cutting-edge automated data labeling operations, and sophisticated toolkits.

A huge workforce (1,000+ data labelers on any particular project) is needed to achieve exceptionally high quality (99%+) on a large dataset. It is challenging to scale to this volume at this quality using crowdsourcing platforms and internal teams. However, these companies can also be expensive and, if they are not acting as a trusted advisor, can convince you to label more data than you may need for a given application.

Pros: Excellent quality, low cost, and technical know-how; The best data labeling businesses has certifications like SOC2 and HIPAA, which are pertinent to their industry.

Cons: Give up authority over the labeling procedure; Need a reliable partner to handle sensitive data that possesses the necessary credentials

You must locate a data labeling platform after deciding who will label your data. Here, one can choose from a variety of approaches, such as constructing internally, utilizing open-source tools, or utilizing paid labeling services.

Open Source Tools

With minor restrictions on commercial use, these tools are available for free to anybody. These resources are excellent for testing early commercial AI applications, working on personal projects, and learning about and developing machine learning and AI. These tools are free, but the cost is that they lack some of the sophistication and scalability of other paid systems. These free tools might not support all of the label kinds covered in this article.

Many excellent open source alternatives might not be included in this list, which is intended to be representative but not exhaustive.

• CVAT: CVAT is a free, open-source web-based data labeling tool that was first created by Intel. Numerous common label shapes are supported by CVAT, such as cuboids, polygons, and rectangles. CVAT is a collaboration tool that works well for smaller or more beginner projects. Nevertheless, the web version's collaboration features are less appealing because it only allows 500 MB of data and 10 projects per person. CVAT can be accessed locally to circumvent these limitations on data.
• LabelMe: LabelMe is an open-source, free data-labeling tool developed by CSAIL that facilitates community participation on computer vision research datasets. After installing the tool, you can label own data and contribute to other projects by labeling public datasets. Labelme is quite limited compared to CVAT, and the web version no longer accepts new accounts.
• Stanford Core NLP: Stanford's CoreNLP is a feature-rich open-source platform for natural language processing and NLP labeling that includes text processing, Named Entity Recognition (NER), linking, and other features.

In-house Tools

Some major firms choose to build their own tools in-house to have more control over their ML pipelines. Which functionality to develop to serve your desired use cases and overcome your particular issues is entirely up to you. Nevertheless, this strategy is expensive, and in order to stay current with emerging technologies, these tools will need to be updated and maintained.

Commercial Platforms

Commercial platforms help you expand by providing expert labeling workforces, specialized support, and top-notch tools. They can also offer advice on machine learning and labeling best practices. Providing support to a large number of clients raises the standard of the platforms for all users, giving you access to cutting-edge features that open-source or in-house labeling systems would not offer.

The greatest labeling infrastructure to speed up your team, labeling tools to support any use case, and orchestration to maximize worker performance are all provided by Refonte.Ai Studio, the top commercial platform in the market. Analyze, track, and enhance the quality of your data with ease.

Bounding boxes, the most often used and basic type of data label, are rectangular boxes that show where an object is in an image or video.

A rectangular box is drawn over an object of interest, like a street sign or car, by data labelers. The X and Y coordinates of the object are defined by this box.

In order to save computer resources and improve the accuracy of object detection, machine learning models can extract certain object properties from a more precise feature set by "bounding" an item with this kind of label.

The practice of classifying items in an image based on where they are located is called object detection. Then, these X and Y coordinates can be exported in a format that is machine-readable, such JSON.

Typical Bounding Box Applications:

• Autonomous driving and robotics to detect objects such as cars, people, or houses
• Identifying damage or defects in manufactured objects
• Household object detection for augmented reality applications
• Anomaly detection in medical diagnostic imaging

Best Practices:

• Hug the border as tightly as possible. Accurate labels will capture the entire object and match the edges as closely as possible to the object's edges to reduce confusion for your model.
• Avoid item overlap. Due to IoU, bounding boxes work best when there is minimal overlap between objects. If objects overlap significantly, using polygon or segmentation annotations may be better.
• Object size: Smaller objects are better suited for bounding boxes, while larger objects are better suited for instance segmentation. However, annotating tiny objects may require more advanced techniques.
• Avoid Diagonal Lines: Bounding boxes perform poorly with diagonal lines such as walkways, bridges, or train tracks as boxes cannot tightly hug the borders. Polygons and instance segmentation are better approaches in these cases.

Object classification means applying a label to an entire image based on predefined categories, known as classes. Labeling images as containing a particular class such as "Dog," "Dress," or "Car" helps train an ML model to accurately predict objects of the same class when run on new data.

Typical Classification Applications:

• Activity Classification
• Product Categorization
• Image Sentiment Analysis
• Hot Dog vs. Not Hot Dog

Best Practices:

• Create clearly defined, easily understandable categories that are relevant to the dataset.
• Provide sufficient examples and training to your data labelers so that the requirements are clear and ambiguity between classes is minimized.
• Create benchmark tests to ensure label quality.

Cuboids are 3-dimensional labels that identify the width, height, and depth of an object, as well as the object's location.

Data labelers draw a cuboid over the object of interest such as a building, car, or household object, which defines the object's X, Y, and Z coordinates. These coordinates are then output in a machine-readable format such as JSON.

Cuboids are crucial for applications like indoor robotics, autonomous driving, and 3D room planners because they allow models to accurately comprehend an object's position in 3D space. It is also easier to comprehend a scene as a whole when these objects are reduced to geometric primitives.

Typical Cuboid Applications:

• Develop prediction and planning models for autonomous vehicles using cuboids on pedestrians and cars to determine predicted behavior and intent.
• Indoor objects such as furniture for room planners
• Picking, safety, or defect detection applications in manufacturing facilities

Best Practices:

• Capture the corners and edges accurately. Like bounding boxes, ensure that you capture the entire object in the cuboid while keeping the label as tight to the object as possible.
• Avoid Overlapping labels where possible. Clean, non-overlapping cuboid data annotations will help your model improve object predictions and localizations in 3D space.
• Axis alignment is critical. Ensure that the alignment of your bounding boxes is on the same axis for objects of the same class.
• Remember the intrinsics of your camera. When objects are not in the same place as the camera in the future, applying cuboids without knowing the location of the camera will result in inaccurate predictions. A "true" cuboid's front face probably won't be a perfect 90-degree rectangle, especially if it isn't directly facing the camera. In addition, the top and bottom edges of the right side of the above annotation are parallel, although the edges of a cuboid parallel to the ground should converge to the horizon.
• Pair 2D Data with 3D Depth Data such as LiDAR.2D images inherently lack depth information, so pairing your 2D data with 3D depth data such as LiDAR will yield the best results for applications dependent on depth accuracy. See the section below on 3D Sensor fusion for more information on this topic.

The process of merging data from several sensors to account for the shortcomings of each sensor is known as three-dimensional sensor fusion. Present machine learning models cannot understand complete scenes from 2D photos alone. It is difficult to estimate depth from a 2D image, and depending solely on 2D images is problematic due to occlusion and a narrow field of view. Some methods of autonomous driving only use cameras, while a more reliable method uses 3D systems to enhance 2D systems with sensors like radar and LiDAR to overcome the limits of 2D.

LiDAR (Light Detection and Ranging) is a technique that uses a laser to measure an object's distance in order to calculate an object's depth and produce three-dimensional (3D) images of the scene.

Radio waves are used by radar, also known as radio detection and range, to measure an object's radial velocity, angle, and distance.

An interactive 3D sensor fused scene is shown in this demo, and a high-level overview of a scenario similar to this one is shown in the video below.

Typical 3D Sensor Fusion Applications

• Autonomous Vehicles
• Geospatial and mapping applications
• Robotics and automation

Best Practices:

• Ensure that your data labeling platform is calibrated to your sensor intrinsics (or better yet, ensure that your tooling is sensor agnostic) and supports different lens and sensor types, for example, fisheye and panoramic cameras.
• Look for a data labeling platform that can support large scenes, ideally with support for infinitely long scenes.
• Ensure that object tracking is consistent throughout a scene, even when an object leaves and returns to the scene.
• Include attribute support for understanding correlations between objects, such as truck cabs and trailers.
• Leverage linked instance IDs describing the same object across the 2D and 3D modalities.

Oval data labels called ellipses are used to indicate where items are located in an image. When data labelers come upon an interesting thing, such wheels, faces, eyes, or fruit, they write an ellipse label on it. The object's location in two dimensions is specified by this annotation. To properly specify the location of the ellipse, the X and Y coordinates of its four extremal vertices can then be exported in a machine-readable format, like JSON.

Applications

• Face Detection
• Medical Imaging Diagnosis
• Wheel Detection

Best Practices:

• The data that needs to be labeled should be circular or oval; in other words, labeling rectangular boxes with ellipses when a bounding box will work better is not beneficial.
• When there is a large degree of overlap for bounding boxes or when items are closely spaced or obscured, like in fruit bunches, use ellipses. Ellipses can give your model a more focused geometry by closely hugging the edges of these items.

Roadway markers and other linear items are identified by lines. The vertices of a line are defined by the lines that data labelers draw over areas of interest. Adding lines to photos can help you teach your model to recognize boundaries more precisely. The line vertices' X and Y coordinates can then be exported in JSON.

Typical Lines Applications

• Label roadway markers with straight or curved lines for autonomous vehicles
• Horizon lines for AR/VR applications
• Define boundaries for sporting fields

Best Practices:

• Label only the lines that matter most to your application.
• Match the lines to the shape of the lines in the image as closely as possible.
• Depending on the use case, it could be important for lines not to intersect.
• Center the line annotation within the line in the image to improve model performance.

The geographic placements of points in an image are used to highlight significant characteristics of an object. Each location of interest is marked with a point by data labelers, which also indicates the location's X and Y coordinates. These ideas might be connected to one another, as in the case of labeling the wrist, elbow, and shoulder of a human being in order to indicate the major moving components of an arm. These labels help machine learning models more accurately determine pose estimations or detect essential features of an object.

Typical Points Applications

• Pose estimation for fitness or health applications or activity recognition
• Facial feature points for face detection

Best Practices:

• Label only the points that are most critical to your application. For instance, if you are building a face detection application, focus on labeling salient points on the eyes, nose, mouth, eyebrows, and the outline of the face.
• Group points into structures (hand, face, and skeletal keypoints), and the labeling interface should make it efficient for taskers to visualize the interconnections between points in these structures.

Although data labelers can quickly and easily create bounding boxes, they can leave significant gaps surrounding objects and are not accurate in mapping to irregular forms. When employing bounding boxes and polygons, accuracy and efficiency are trade-offs. Bounding boxes can easily create a machine learning model with enough accuracy for many applications. On the other hand, certain applications need higher polygon accuracy at the risk of an expensive and ineffective annotation.

By clicking on pertinent locations on an object of interest, data labelers create a polygon shape over the object to complete a totally connected annotation. The polygon's vertices are defined by these points. Next, a JSON output of these vertices' X and Y coordinates is produced.

Typical Polygons Applications

• Irregular objects such as buildings, vehicles, or trees for autonomous vehicles
• Satellite imagery of houses, pools, industrial facilities, planes, or landmarks
• Fruit detection for agricultural applications

Best Practices:

• Special attention must be given to objects that have holes in them or that split into several polygons as a result of occlusion (a automobile behind a tree, for example). Take each hole's area and subtract it from the object.
• Keep adjacent polygons from slightly overlapping one another.
• To make sure that you set dots close to the edges of each object, zoom in close to each one.
• Keep an eye out for curved edges and be sure to add additional vertices to 'smooth' them as much as possible.
• Make effective use of the Auto Annotate Polygon tool to label objects. Quickly and automatically create high-precision polygon annotations by using an initial, approximative bounding box to highlight particular items of interest.

There are three primary forms of segmentation labels: panoptic, instance, and semantic. Segmentation labels correspond to pixel-wise labeling on an image.

Semantic Segmentation

Give each pixel in an image a class corresponding to the object it represents, for as a person, a car, or some vegetation. This procedure, known as "dense prediction," is laborious and time-consuming.

You cannot discriminate between distinct objects of the same class using semantic segmentation (for more on this, see instance segmentation).

Instance segmentation

Label every pixel of each distinct object of an image. Unlike semantic segmentation, instance segmentation distinguishes between separate objects of the same class (i.e., identifying car 1 as separate from car 2).

Panoptic Segmentation

Semantic and instance segmentation are used to provide panoptic segmentation. Both an instance label (instance segmentation) and a class label (semantic segmentation) are assigned to each point in an image. Every instance can stand in for other things, including vehicles, people, or spaces, like the road or the sky. Deeper scene knowledge can be achieved with panoptic segmentation since it is more detailed and offers more context than semantic segmentation.

Typical Segmentation Applications

• Autonomous Vehicles and Robotics: Identify pedestrians, cars, trees
• Medical Diagnostic imaging: tumors, abscesses in diagnostic imaging
• Clothing: Fashion retail

Best Practices

• Carefully trace the outlines of each shape to ensure that all pixels of each object are labeled.
• Use ML-assisted tooling like the boundary tool to quickly segment borders and objects of interest.
• After segmenting borders, use the flood fill tool to fill in and complete segmentation masks quickly.
• Use active tools like Autosegment to increase the efficiency and accuracy of your labelers.

Explore Coco-Stuff on Nucleus for a large collection of data with segmentation labels!

You can apply many of the same labels to images and videos, but there are some special considerations for video labeling.

Temporal linking of labels

Video annotations add the dimension of time to the data fed to your models, so your model needs a way to understand related objects and labels between frames.

Video annotations add the dimension of time to the data fed to your models, so your model needs a way to understand related objects and labels between frames.

Manually tracking objects and adding labels through many video frames is time and resource intensive. You can leverage several techniques to increase the efficiency and accuracy of temporally linked labels.

• First, you can leverage video Interpolation to interpolate between frames of your video to smooth out the images and labels to make it easier to track labels through the video.
• You should also look for tools that automatically duplicate annotations between frames, minimizing human intervention needed to correct for misaligned labels.
• If you are working with videos, ensure that the tools you are using can handle the storage capacity of the video file and can stitch together hour-long videos so that you retain the same context no matter how long the video.

In videos, objects may leave the camera view and may return at a later time. Leverage tools that enable you to track these objects automatically or make sure to remember to annotate these objects with the same unique ids.

Multimodal

Multimodal machine learning attempts to understand the world from multiple, related modalities such as 2D images, audio, and text.

Combining multiple labeling types such as human keypoints, bounding boxes, and transcribed audio with entity recognition all connected in rich scenes.

Typical Multimodal Applications

• AR/VR full scene understanding Video/GIF/Image (Object Detection + Human Keypoints + Audio Transcription + Entity Recognition)
• Sentiment analysis by combining video gestures and voice data

Best Practices

• Incorporate temporal linking to ensure that models fully understand the entire breadth of each scene.
• Identify which modalities are best suited for your application. For instance, if you are working on sentiment analysis for AR/VR applications, you will want to consider not only 2D video object or human keypoint labels but also audio transcription and entity recognition in addition to sentiment classification so that you have a rich understanding of the entire scene and how the individuals in the scene contribute towards a particular sentiment (i.e., if the person is yelling and gesturing wildly you can determine the sentiment is "upset").
• Include Human in the loop to ensure consistency across modalities. Assign complex scenes to only the most experienced taskers.

Digitally created data that imitates real-world data is called synthetic data. Often, artists use graphics computer tools to generate this data, or models such as Neural Radiance Fields (NERFs), Variational Autoencoders (VAEs), and Generative Adversarial Networks (GANs) are used programmatically to generate this data. Perfect ground truth labels are automatically included in synthetic data; no extra human labeling is needed.

Typical Synthetic Data Applications

• Digital Humans for autonomous vehicles and robotics, particularly in long-tail edge cases such as pedestrians walking on shoulders
• Digital humans for fitness and health applications. Getting enough real-world human pose data is difficult and expensive, whereas synthetic data is relatively easy to generate and cheaper.
• Manufacturing defect detection

Best Practices:

• If using a mix of Synthetic and real-world data, ensure that the labels of your real data are as accurate as possible. Synthetic data generates perfectly accurate labels, so any label inaccuracies in your real data will degrade your model's predictive capabilities.
• Leverage synthetic data for data that is difficult to collect due to privacy concerns, rare edge cases, to avoid bias, or prohibitively expensive data collection and labeling methods.
• Integrate synthetic data with your existing data pipelines to maximize your ROI
• Curate your data using best-in-class tools to ensure that you are surfacing the edge cases for which you need more data. Ideally, your dataset curation tool will also integrate into your labeling pipelines.