Active Learning-Based Human-in-the-Loop Object Detection for Scalable Data Annotation

Lightly adapted for web publication from my 2021 M.Sc. thesis work.

Summary

This work explored a human-in-the-loop active learning framework for deep object detection.

The objective was to reduce manual annotation effort by selecting only the most informative images for human labelling. Instead of annotating the full dataset, the system used an object detection model to score unlabelled images by uncertainty, select high-value samples, send them to human annotators, and retrain the model incrementally.

The broader technical focus was data efficiency: improving model performance while reducing redundant annotation work.

Key Results

In the experimental setup, the active learning pipeline:

• reduced the labelled training set from 12,000 images to 3,100 images;
• reduced estimated annotation time from 144,000 seconds to 37,200 seconds;
• achieved 0.7541 test mAP, compared with 0.7010 mAP for the standard baseline trained on the full dataset.

The result does not imply that smaller datasets are generally better. It shows that, in this setup, uncertainty-selected samples carried more useful training signal than a larger passively annotated dataset with redundant samples.

Motivation

Object detection workflows are annotation-heavy. Each image may contain multiple objects, and each object requires spatial labelling through bounding boxes. In practical data collection settings, especially video or sequential image capture, many frames are visually similar and provide limited additional training value.

The main problem was to reduce unnecessary annotation while preserving, or improving, model performance.

Active learning had been widely studied for image classification, but object detection introduced additional complexity:

1. Multiple predictions per image
   Object detection models generate many bounding boxes per image, each with objectness, class confidence, and spatial coordinates.
2. Uncertainty aggregation
   Bounding-box-level uncertainty must be converted into a single image-level score for sample selection.
3. Redundant visual data
   Consecutive frames or similar images can dominate the annotation queue unless uncertainty is combined with diversity-aware sampling.

System Overview

The proposed system used a closed-loop active learning architecture:

[Unlabelled Raw Data]
        |
        v
[Object Detection Model Inference]
        |
        v
[Uncertainty Scoring]
        |
        +----> [Certain / Low-Value Samples Discarded]
        |
        v
[Selected Uncertain Samples]
        |
        v
[Human Annotation]
        |
        v
[Incremental Model Training]
        |
        v
[Updated Model]

The system combined transfer learning, object detection, uncertainty-based sample selection, human annotation, and incremental model training.

Detection Backbone

The object detection backbone was based on YOLOv3 with a Darknet-53 architecture.

YOLOv3 was selected due to its inference speed and suitability for practical vision systems. It performs detection across three scales:

• 13 × 13 for larger objects;
• 26 × 26 for medium-sized objects;
• 52 × 52 for smaller objects.

For a 416 × 416 input image with three anchor boxes per grid cell, this produces:

(13 × 13 × 3) + (26 × 26 × 3) + (52 × 52 × 3) = 10,647 candidate bounding boxes

Each prediction contains bounding box coordinates, an objectness score, and class confidence scores.

Uncertainty Scoring

The sample selection process had three steps.

1. Objectness Filtering

Candidate bounding boxes with low objectness scores were removed before uncertainty calculation.

2. Margin of Confidence

For each remaining bounding box, uncertainty was calculated using the difference between the top two class probabilities:

uncertainty = 1 - (probability of top class - probability of second class)

A smaller margin between the top two classes indicated higher uncertainty.

3. Image-Level Aggregation

Bounding-box-level uncertainty scores were aggregated into an image-level score. Three strategies were evaluated:

• maximum aggregation;
• average aggregation;
• sum aggregation.

In this experiment, sum aggregation produced the most stable results for multi-object images because it reflected both the number of uncertain objects and their individual uncertainty scores.

Experimental Setup

The experiments used the PASCAL VOC2012 dataset.

Three setups were compared:

1. Standard Baseline
   A supervised model trained on the full annotated dataset.
2. Random Selection Baseline
   A model trained on a randomly selected subset of approximately 3,000 images.
3. Active Selection Model
   A model trained incrementally using uncertainty-selected samples.

The active selection model used margin-of-confidence uncertainty scoring with sum aggregation. Data augmentation included brightness, contrast, saturation, and horizontal flipping.

Results

Random selection reduced annotation effort but caused a large performance drop. Active selection reduced annotation effort while improving test mAP in this setup.

The result suggests that labelled dataset value depends not only on volume, but also on sample informativeness.

Limitations

The experiments were conducted on a benchmark dataset rather than a production video stream.

Uncertainty sampling alone can also over-select visually similar samples. In sequential image data, similar frames may receive similar uncertainty scores and fill the annotation queue with redundant examples.

A production-grade version would need to combine uncertainty scoring with:

• embedding-based diversity sampling;
• core-set selection;
• clustering;
• class-aware sampling;
• temporal deduplication for video streams.

Relevance

The main technical principle of this work was selective feedback: an AI system should identify which samples are uncertain, informative, and worth human review.

In early Co-one, this principle was applied to data annotation workflows. Later, similar feedback-loop thinking became relevant to model evaluation and AI agent reliability: identifying uncertain outputs, routing failures for review, and improving systems through targeted feedback.

Closing

This thesis work focused on object detection and annotation efficiency. Its broader relevance was the design of AI systems that use human feedback selectively instead of treating all data or outputs equally.