Label Quality Metrics

Label Quality Metrics operate on the geometries of objects like bounding boxes, polygons, and polylines.

Title	Metric Type	Data Type
Annotation Duplicates - Ranks annotations by how likely they are to represent the same object.	image	bounding box, polygon, rotatable bounding box
Annotation closeness to image borders - Ranks annotations by how close they are to image borders.	image	bounding box, point, polygon, polyline, rotatable bounding box, skeleton
Detect Occlusion in Video - Tracks objects and detect outliers.	sequence	bounding box, rotatable bounding box
Frame object density - Computes the percentage of image area that's occupied by objects.	image	bounding box, polygon, rotatable bounding box
Image-level Annotation Quality - Compares image classifications against similar images.	image	radio
Inconsistent Object Classification and Track IDs - Looks for overlapping objects with different classes (across frames).	sequence	bounding box, polygon, rotatable bounding box
Missing Objects and Broken Tracks - Identifies missing objects and broken tracks based on object overlaps.	sequence	bounding box, polygon, rotatable bounding box
Object Annotation Quality - Compares object annotations against similar image crops.	image	bounding box, polygon, rotatable bounding box
Object Area - Absolute - Computes object area in amount of pixels.	image	bounding box, polygon, rotatable bounding box
Object Area - Relative - Computes object area as a percentage of total image area.	image	bounding box, polygon, rotatable bounding box
Object Aspect Ratio - Computes aspect ratios of objects.	image	bounding box, polygon, rotatable bounding box
Object Count - Counts number of objects in the image.	image	bounding box, checklist, point, polygon, polyline, radio, rotatable bounding box, skeleton, text
Polygon Shape Similarity - Ranks objects by how similar they are to their instances in previous frames.	sequence	polygon
Random Values on Objects - Assigns a random value between 0 and 1 to objects.	image	bounding box, polygon, rotatable bounding box
Shape outlier detection - Calculates potential outliers by polygon shape.	image	polygon

Annotation Duplicates

Ranks annotations by how likely they are to represent the same object.

Jaccard similarity coefficient is used to measure closeness of two annotations.

Implementation on GitHub

Annotation closeness to image borders

This metric ranks annotations by how close they are to image borders.

Implementation on GitHub

Detect Occlusion in Video

This metric collects information related to object size and aspect ratio for each track and find outliers among them.

Implementation on GitHub

Frame object density

Computes the percentage of image area that's occupied by objects.

Implementation on GitHub

Image-level Annotation Quality

This metric creates embeddings from images. Then, these embeddings are used to build nearest neighbor graph. Similar embeddings' classifications are compared against each other.

Implementation on GitHub

Inconsistent Object Classification and Track IDs

This algorithm looks for overlapping objects in consecutive frames that have different classes. Furthermore, if classes are the same for overlapping objects but have different track-ids, they will be flagged as potential inconsistencies in tracks.

Example 1:

      Frame 1                       Frame 2
┌───────────────────┐        ┌───────────────────┐
│                   │        │                   │
│  ┌───────┐        │        │  ┌───────┐        │
│  │       │        │        │  │       │        │
│  │ CAT:1 │        │        │  │ DOG:1 │        │
│  │       │        │        │  │       │        │
│  └───────┘        │        │  └───────┘        │
│                   │        │                   │
└───────────────────┘        └───────────────────┘

Dog:1 will be flagged as potentially wrong class, because it overlaps with CAT:1.

Example 2:

      Frame 1                       Frame 2
┌───────────────────┐        ┌───────────────────┐
│                   │        │                   │
│  ┌───────┐        │        │  ┌───────┐        │
│  │       │        │        │  │       │        │
│  │ CAT:1 │        │        │  │ CAT:2 │        │
│  │       │        │        │  │       │        │
│  └───────┘        │        │  └───────┘        │
│                   │        │                   │
└───────────────────┘        └───────────────────┘

Cat:2 will be flagged as potentially having a broken track, because track ids 1 and 2 doesn't match.

Implementation on GitHub

Missing Objects and Broken Tracks

Identifies missing objects by comparing object overlaps based on a running window.

Case 1: If an intermediate frame (frame $i$) doesn't include an object in the same region, as the two surrounding frames ($i-1$ and $i+1$), it is flagged.

      Frame i-1                     Frame i                      Frame i+1
┌───────────────────┐        ┌───────────────────┐        ┌───────────────────┐
│                   │        │                   │        │                   │
│  ┌───────┐        │        │                   │        │  ┌───────┐        │
│  │       │        │        │                   │        │  │       │        │
│  │ CAT:1 │        │        │                   │        │  │ CAT:1 │        │
│  │       │        │        │                   │        │  │       │        │
│  └───────┘        │        │                   │        │  └───────┘        │
│                   │        │                   │        │                   │
│                   │        │                   │        │                   │
└───────────────────┘        └───────────────────┘        └───────────────────┘

Frame $i$ will be flagged as potentially missing an object.

Case 2: If objects of the same class overlap in three consecutive frames ($i-1$, $i$, and $i+1$) but do not share object hash, they will be flagged as a potentially broken track.

      Frame i-1                     Frame i                      Frame i+1
┌───────────────────┐        ┌───────────────────┐        ┌───────────────────┐
│                   │        │                   │        │                   │
│  ┌───────┐        │        │  ┌───────┐        │        │  ┌───────┐        │
│  │       │        │        │  │       │        │        │  │       │        │
│  │ CAT:1 │        │        │  │ CAT:2 │        │        │  │ CAT:1 │        │
│  │       │        │        │  │       │        │        │  │       │        │
│  └───────┘        │        │  └───────┘        │        │  └───────┘        │
│                   │        │                   │        │                   │
│                   │        │                   │        │                   │
└───────────────────┘        └───────────────────┘        └───────────────────┘

CAT:2 will be marked as potentially having a wrong track id.

Implementation on GitHub

Object Annotation Quality

This metric transforms polygons into bounding boxes and an embedding for each bounding box is extracted. Then, these embeddings are compared with their neighbors. If the neighbors are annotated differently, a low score is given to it.

Implementation on GitHub

Object Area - Absolute

Computes object area in amount of pixels.

Implementation on GitHub

Object Area - Relative

Computes object area as a percentage of total image area.

Implementation on GitHub

Object Aspect Ratio

Computes aspect ratios ($width/height$) of objects.

Implementation on GitHub

Object Count

Counts number of objects in the image.

Implementation on GitHub

Polygon Shape Similarity

Ranks objects by how similar they are to their instances in previous frames based on Hu moments. The more an object's shape changes, the lower its score will be.

Implementation on GitHub

Random Values on Objects

Uses a uniform distribution to generate a value between 0 and 1 to each object

Implementation on GitHub

Shape outlier detection

Computes the Euclidean distance between the polygons' Hu moments for each class and the prototypical class moments.

Implementation on GitHub