Object Detection Algorithms

Object detection answers two questions: what objects are present, and where are they? A detector normally returns a set of class labels, confidence scores, and bounding boxes. Over several decades, the field has moved from hand-built templates scanned across an image to neural networks that predict many objects in one pass, and then to transformers and vision-language models that can search for categories described in text.

No detector is universally best. A large open-vocabulary model may recognize unusual objects in a photograph, while a compact closed-set model may be far better for a 60-frame-per-second camera. For video, per-frame accuracy is only part of the problem: results must arrive before their frames become stale and object identities should remain stable over time.

Detection and Neighboring Tasks #

Image classification assigns labels to an entire image. It can say that a dog is present, but not where the dog is or whether there are three dogs.

Object detection returns one record per detected instance:

{class: "dog", confidence: 0.94, box: [x_min, y_min, x_max, y_max]}

Semantic segmentation assigns a class to each pixel. Instance segmentation also separates different objects of the same class. Matting estimates fractional opacity at soft edges. A detector’s rectangle is therefore useful for locating or selecting an object, but is insufficient for precise background removal.

Tracking links detections across video frames and assigns persistent identities. Detection can be the first stage of tracking, but a detector alone does not know whether the person in frame 100 is the same person seen in frame 1.

Modern systems blur these boundaries. A shared visual encoder may feed box, mask, pose, and tracking heads, while a promptable model may accept text or an example image instead of a fixed class list. The output and evaluation criterion still determine which problem the system actually solves.

From Sliding Windows to Deformable Parts #

Early detectors searched an image with a template. A fixed-size classifier examined every possible location; an image pyramid repeated the search at different scales. This sliding-window design is conceptually simple, but dense search is expensive and rigid templates do not handle pose or appearance variation well.

The 2001 Viola-Jones detector made real-time face detection practical. It used integral images to evaluate Haar-like contrast features quickly, AdaBoost to select useful features, and a cascade that rejected easy negative windows early. It remains attractive for constrained frontal-face or rigid-object tasks on a CPU. Its hand-designed features and limited pose tolerance are poor matches for arbitrary objects in clutter.

Histograms of Oriented Gradients (HOG) represented local edge directions and paired them with a linear support vector machine. HOG was robust enough to become a strong pedestrian-detection baseline, but still required a scale pyramid and separate classifier evaluation over many windows.

Deformable Part Models (DPMs) represented an object as a coarse root filter plus movable part filters. A latent SVM learned both appearance and deformation penalties, allowing a person’s limbs or a vehicle’s components to move relative to the whole. DPMs improved pose handling and made structured object models explicit, but inference over pyramids and parts was expensive. Their accuracy was overtaken when learned convolutional features became practical.

The Convolutional Detector Era #

Deep detectors replaced hand-engineered features with features learned from data. Their central design question was where to spend computation.

Two-Stage Detectors #

R-CNN generated about two thousand category-independent region proposals, ran a convolutional network on every proposal, then classified and adjusted each box. It demonstrated the value of deep features, but repeated network evaluation made it slow and its multi-stage training pipeline was cumbersome.

Fast R-CNN ran the convolutional backbone once for the whole image and pooled features for each proposed region. Faster R-CNN then replaced the external proposal algorithm with a learned Region Proposal Network. The resulting two stages are:

1. propose locations that may contain objects; and
2. classify and refine those proposals.

Two-stage detectors tend to offer strong localization and small-object accuracy because the second stage spends focused computation on each candidate. Their proposal processing, irregular workloads, and per-region heads usually make them slower and harder to batch than a compact one-stage model.

Feature Pyramid Networks (FPNs) added top-down, multi-scale features so that small and large objects could be detected at suitable resolutions. This idea became important across both two-stage and one-stage architectures. Mask R-CNN extended Faster R-CNN with a parallel mask head; it is an instance-segmentation model, although its boxes can also be evaluated as detections.

One-Stage Detectors #

One-stage models predict classes and boxes densely without a separate proposal-and-classification phase. YOLO framed detection as a single regression problem over the image. The original model was dramatically faster than proposal-based contemporaries, although it made more localization errors and struggled with groups of small objects. Later YOLO families adopted multi-scale features, better losses, stronger backbones, and deployment-oriented implementations; the shared idea matters more than a version number.

SSD predicted boxes from several feature-map resolutions, matching default boxes of different shapes to objects. This provided a direct speed-accuracy tradeoff and worked well on accelerators. Default boxes, also called anchors, require choices about scales, aspect ratios, and label assignment.

Dense one-stage detectors face a severe imbalance: most candidate locations are background. RetinaNet showed that focal loss could down-weight easy negatives and focus training on hard examples, closing much of the quality gap with two-stage detectors.

Anchor-free designs removed predefined box templates. FCOS predicts distances from foreground feature locations to four box sides, while CenterNet represents objects by center points and regresses their size. These designs simplify configuration and can reduce the number of candidates, but they still need rules for assigning overlapping objects and resolving duplicate predictions.

After inference, many convolutional detectors use non-maximum suppression (NMS). The highest-scoring box is kept and strongly overlapping alternatives are removed. NMS is efficient, but its overlap threshold creates a tradeoff: suppress too aggressively and nearby objects disappear; suppress too weakly and one object produces duplicates. Crowds are particularly difficult.

Transformers and Set Prediction #

DETR introduced an end-to-end transformer detector. A convolutional backbone produces image features, a transformer reasons globally about them, and a fixed set of learned object queries predicts a set of boxes. Bipartite matching assigns predictions to ground truth during training, which removes anchor design and conventional NMS.

The clean formulation came with costs. Original DETR converged slowly, used substantial compute, and had difficulty with small objects because attention operated on relatively coarse features. Deformable DETR attended to a small set of sampling points across multiple feature levels, improving convergence and small-object detection while reducing attention cost.

Later DETR descendants improved query initialization, denoising training, matching, and feature fusion. DINO is a representative quality-oriented design. RT-DETR redesigned the encoder and query selection for real-time use and exposed model scaling without relying on NMS. In practice, the boundary between “CNN detector” and “transformer detector” is not absolute: efficient systems commonly use convolutional backbones, multi-scale pyramids, and transformer-style decoders together.

Transformers are attractive when global context, clean end-to-end training, or a shared multimodal representation matters. Compact one-stage convolutional models remain competitive where kernels are highly optimized, memory is tight, and predictable edge-device latency matters most.

Open-Vocabulary and Prompted Detection #

Traditional closed-set detection recognizes only the classes defined during training. Adding a new class generally requires annotated examples and retraining or fine-tuning.

Open-vocabulary detection aligns regions with language representations. OWL-ViT adapts a vision transformer trained on image-text pairs so that text embeddings act as classification weights. GLIP reformulates detection as phrase grounding and trains jointly on detection and grounding data. Grounding DINO combines grounded language features with a DETR-like detector and can return boxes associated with words in a prompt.

This changes the interface from a fixed class index to a query:

"red fire extinguisher . exit sign . person wearing a hard hat"

The advantages are flexibility, long-tail coverage, and rapid prototyping without a task-specific output head. The costs are larger encoders, text processing, more variable latency, and prompt-sensitive semantics. A model may understand a phrase but localize it poorly, confuse attributes with objects, or return different results for synonyms. A broad pretrained vocabulary also does not guarantee reliable domain knowledge for medical, industrial, or safety-critical use.

As of 2026, the cutting edge is not one architecture. It includes stronger real-time DETR variants, open-vocabulary detectors distilled for edge deployment, detectors that accept text or visual exemplars, and unified vision models that can emit boxes, masks, or language. Two public examples are the 2025 RF-DETR preprint, which uses architecture search to explore real-time DETR configurations, and RT-DETRv4, which distills supervision from vision foundation models into lightweight detectors. Their reported latency and accuracy are author results for particular artifacts and hardware, not deployment-independent rankings.

Quality claims depend on the dataset, class vocabulary, input resolution, model scale, and hardware. A specialized detector can still outperform a foundation model on a narrow, stable task while using a fraction of the compute.

Static-Image Tradeoffs #

Static images allow arbitrary delay, multi-scale inference, tiling, test-time augmentation, and large models. The system can optimize final quality rather than causality.

Resolution is one of the strongest controls. Lowering it reduces pixel and feature-map work but erases small objects. Raising it improves detail while increasing latency and memory roughly with image area, and transformer attention may scale less favorably unless it is sparse or deformable.

Tiling a large aerial, document, or microscopy image preserves tiny objects. It also repeats backbone work, increases latency, and creates duplicate boxes at tile boundaries. Overlap and a global merge step are necessary. Multi-scale inference has similar costs and should be justified by measured improvement on the objects that matter.

Quantization, pruning, distillation, and smaller backbones reduce deployment cost. FP16 is often a low-risk accelerator optimization; INT8 can improve speed and memory further but needs representative calibration or quantization-aware training. Detection heads and NMS must be included in benchmarks because an optimized backbone can leave post-processing as the bottleneck.

Detection in Video Streams #

Running an image detector independently on every frame is a valid baseline, but it ignores two facts: adjacent frames are redundant, and a result delivered too late is no longer useful.

Tracking by Detection #

The most common video architecture detects objects and then associates them over time:

frame -> detector -> boxes -> motion/appearance association -> tracks

SORT uses a Kalman filter to predict box motion and the Hungarian algorithm to match new detections. It is fast and simple, but identity switches occur during occlusion or missed detections. Deep SORT adds learned appearance embeddings, helping distinguish objects whose paths cross at the cost of another model and domain-sensitive features.

ByteTrack associates high-confidence detections first, then uses low-confidence detections to recover partially occluded objects rather than discarding them. BoT-SORT combines motion, appearance, and camera-motion compensation. These systems can dramatically improve track continuity, but they cannot recover an object that the detector never observes for long enough, and their thresholds couple detector calibration to tracker behavior.

Joint detection-and-tracking models share features or predict object queries across frames. They can learn temporal cues end to end and amortize work, but are harder to train, maintain state, and reset correctly. Tracking by detection remains modular: the detector or tracker can be upgraded independently, and failures are easier to inspect.

Temporal Detection #

Video detectors can reuse information rather than recomputing every frame:

• run a full detector on keyframes and propagate boxes or features with optical flow;
• aggregate features from nearby frames to strengthen blurred or occluded objects;
• maintain recurrent features or object queries causally;
• use a small model on every frame and invoke a larger model when confidence drops; or
• adapt resolution and frame-skipping rates to motion and system load.

Temporal aggregation can improve quality because a clear neighboring view rescues a blurred current frame. Offline processing may use both past and future frames. A live stream cannot wait indefinitely for future evidence, so it needs a causal model or an explicit look-ahead delay.

Propagation saves compute but drifts during fast motion, occlusion, camera cuts, or deformation. Keyframes re-anchor the system but produce latency spikes unless scheduled carefully. Stateful models need resets after scene cuts, long gaps, timestamp discontinuities, or resolution changes; otherwise old objects leak into a new scene.

Streaming Latency #

A conventional benchmark gives every frame to the model and scores the eventual output. A live application sees a different reality: while the model processes frame \(t\), frames \(t+1\) and \(t+2\) may already have arrived. Streaming perception evaluates the prediction available at each real-world time rather than pretending computation is instantaneous.

At 30 fps, a new frame arrives every 33.3 ms; at 60 fps, every 16.7 ms. If end-to-end processing is slower, an unbounded queue makes detections increasingly stale. A live system should process the newest useful frame, drop obsolete work, or reduce quality under load.

The full path is:

capture/decode -> resize and transfer -> inference -> NMS/association
               -> application logic -> draw/encode/transmit

Model-only frames per second omits decode, memory copies, post-processing, tracking, rendering, encoding, and network jitter. Batching improves server throughput but delays the first frame while a batch forms. Batch size one is usually preferable for interactive video; offline video analytics may batch across frames or streams.

Speed, Quality, and Systems Tradeoffs #

For edge hardware, measure the exact exported graph and runtime. Unsupported operations can fall back to a CPU and erase an architectural advantage. Avoid unnecessary CPU-GPU copies, fuse preprocessing, keep tensors in the accelerator’s native layout, and benchmark after thermal steady state.

For cloud systems, throughput and latency are different objectives. Dynamic batching may reduce cost per image while increasing queue delay. Autoscaling must account for model load time and accelerator availability. Sending cropped regions can save bandwidth, but deciding the crop may itself require local detection and can remove useful context.

Evaluating Detectors and Trackers #

A predicted box is normally matched to a ground-truth box using intersection over union:

Precision measures how many reported detections are correct; recall measures how many real objects were found. Average Precision (AP) summarizes a precision-recall curve. The COCO evaluation reports AP averaged over IoU thresholds from 0.50 through 0.95, plus values for small, medium, and large objects. AP50 is more forgiving than AP75; neither should be confused with classification accuracy.

An aggregate AP can conceal the failure that matters. Report results by class, object size, crowding, occlusion, lighting, motion blur, camera type, and operating confidence threshold. Safety systems also need false negatives per relevant unit of time or distance and calibrated uncertainty, not only leaderboard AP.

Tracking evaluation adds identity. MOTA combines false positives, misses, and identity switches but can overemphasize detection. IDF1 measures identity consistency. HOTA balances detection, association, and localization more explicitly. Report track initialization delay, survival through occlusion, and recovery after camera cuts alongside these metrics.

For deployment, measure:

• median and tail end-to-end latency, not only mean model time;
• delivered frame rate, dropped frames, and prediction age;
• CPU, accelerator, memory, power, and sustained temperature;
• cold-start and state-reset time; and
• accuracy at the chosen resolution, precision, runtime, and confidence threshold.

Comparing paper-reported FPS values is rarely valid unless the hardware, batch size, input size, numerical precision, runtime, preprocessing, and post-processing are identical.

Choosing an Architecture #

For a fixed, narrow task on constrained hardware, start with a compact closed-set one-stage detector and tune resolution from the smallest important object. Choose a two-stage FPN detector when small-object localization justifies latency. Choose a DETR descendant when end-to-end set prediction, global reasoning, or integration with a larger transformer system matters.

Use open-vocabulary detection when users define the search target, classes change frequently, or annotation assistance matters. For a stable production ontology, distilling or fine-tuning a smaller detector may yield better latency and calibration.

For video, first establish a per-frame detector baseline. Add SORT-like association when identities matter, appearance features when occlusions cause switches, and temporal detection only when measurements show that frame redundancy or motion blur justifies the extra state. A live system should select models by worst acceptable latency and prediction age, not by offline throughput.

The practical frontier is hybrid: a fast specialist handles ordinary frames, temporal state supplies continuity, and a larger open-vocabulary or quality model handles unfamiliar queries, difficult keyframes, or offline review. The correct detector is the least expensive system that finds the required objects, at the required sizes and error rates, before its answer becomes stale.

References #

• Viola and Jones, “Rapid Object Detection Using a Boosted Cascade of Simple Features”, 2001.
• Dalal and Triggs, “Histograms of Oriented Gradients for Human Detection”, 2005.
• Felzenszwalb et al., “Object Detection with Discriminatively Trained Part-Based Models”, 2010.
• Girshick et al., “Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation”, 2014.
• Girshick, “Fast R-CNN”, 2015.
• Ren et al., “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks”, 2015.
• Redmon et al., “You Only Look Once: Unified, Real-Time Object Detection”, 2016.
• Liu et al., “SSD: Single Shot MultiBox Detector”, 2016.
• Lin et al., “Feature Pyramid Networks for Object Detection”, 2017.
• Lin et al., “Focal Loss for Dense Object Detection”, 2017.
• Tian et al., “FCOS: Fully Convolutional One-Stage Object Detection”, 2019.
• Zhou, Wang, and Krähenbühl, “Objects as Points”, 2019.
• Carion et al., “End-to-End Object Detection with Transformers”, 2020.
• Zhu et al., “Deformable DETR”, 2021.
• Zhang et al., “DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection”, 2023.
• Zhao et al., “DETRs Beat YOLOs on Real-Time Object Detection”, 2024.
• Robinson et al., “RF-DETR: Neural Architecture Search for Real-Time Detection Transformers”, 2025 preprint.
• Liao et al., “RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Models”, 2025 preprint.
• Minderer et al., “Simple Open-Vocabulary Object Detection with Vision Transformers”, 2022.
• Li et al., “Grounded Language-Image Pre-Training”, 2022.
• Liu et al., “Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection”, 2024.
• Bewley et al., “Simple Online and Realtime Tracking”, 2016.
• Wojke, Bewley, and Paulus, “Simple Online and Realtime Tracking with a Deep Association Metric”, 2017.
• Zhang et al., “ByteTrack: Multi-Object Tracking by Associating Every Detection Box”, 2022.
• Aharon et al., “BoT-SORT: Robust Associations Multi-Pedestrian Tracking”, 2022.
• Li, Wang, and Ramanan, “Towards Streaming Perception”, 2020.