Object Tracking Across Video Frames, Explained

Running a detector on every frame gives you boxes. It does not give you objects. Turning “there is a person here in frame 200 and a person here in frame 201” into “this is the same person” is the association problem, and it is where multi-object tracking is won or lost.

Detection and association are separate jobs

The dominant paradigm is tracking-by-detection: a detector proposes boxes independently on each frame, and a separate stage links boxes across frames into tracks. The separation is deliberate and it is why you can upgrade a detector without touching the tracker. It also means the two components fail in ways that look identical on a video and are fixed by completely different work — a flickering box is a detection recall problem, a box whose number changes is an association problem, and treating the second as the first wastes weeks.

A track is a state machine, not just a list of boxes. A new detection that matches nothing starts a tentative track, promoted to confirmed only after it is matched on several consecutive frames — this is what stops a single false detection creating a permanent object. A confirmed track that goes unmatched is kept alive for a maximum age before deletion, so an object occluded for half a second can be re-acquired with the same identity. Both thresholds are tunable and both are the wrong default for some footage: a long max age on a crowded scene invites identity theft, a short one guarantees fragmentation behind every pillar.

How association is actually computed

SORT, from Bewley and colleagues in 2016, is the minimal version and still the right thing to understand first, because everything since is an addition to it. Its paper is on arXiv. Two components:

• A motion model. Each track carries a Kalman filter with a linear constant-velocity state — box centre, scale, aspect ratio and the velocities of the first three. Aspect ratio is modelled as constant, which is an assumption worth remembering when tracking something that rotates. Before matching, every track is predicted forward one frame.
• An assignment. Build a cost matrix between predicted track boxes and new detections, using one minus IoU as the cost, and solve it optimally with the Hungarian algorithm. Pairs whose IoU falls below a minimum — 0.3 in the reference implementation — are rejected rather than accepted as poor matches.

Optimal assignment is not the same as correct assignment. The Hungarian algorithm returns the cheapest total pairing given the cost matrix. If the cost matrix contains no information distinguishing two candidates, it returns a pairing anyway.

An ID switch, worked

Two people walk towards each other along the same horizontal line. Detected boxes are 40 pixels wide and 100 tall, so each has an area of 4,000 square pixels. Track A’s centre moves +8 pixels per frame, track B’s moves −8.

frame k-1   A centre x = 132   B centre x = 148
Kalman predicts for frame k:
            A -> 140          B -> 140          (identical predictions)

frame k detections:  D1 at x = 136,  D2 at x = 144

IoU(pred 140, D1 at 136):
  spans [120,160] and [116,156], overlap = 156 - 120 = 36 px
  area  = 36 x 100 = 3600
  union = 4000 + 4000 - 3600 = 4400
  IoU   = 3600 / 4400 = 0.818

IoU(pred 140, D2 at 144):  by symmetry, also 0.818

cost matrix          D1      D2
  track A          0.182   0.182
  track B          0.182   0.182

Every entry is equal. Both assignments are optimal, the solver picks one by whatever order it happened to enumerate in, and there is a fifty per cent chance that A leaves the crossing with B’s number. That is an identity switch, and it arises from the cost function containing no information at the one moment it is needed — not from a bug and not from a weak detector.

The same degeneracy explains why ID switches cluster at occlusions, crowd density and camera cuts. After a shot boundary the motion model is meaningless, so a tracker run across an edited video without boundary detection produces garbage associations at every cut. Reset the tracker at each boundary; it costs nothing and removes a whole class of error.

What appearance and low-score boxes fix

Appearance embeddings. DeepSORT adds a small re-identification network producing a unit-norm feature vector per box, and matches on cosine distance in that space, gated by a Mahalanobis distance from the Kalman prediction so that appearance cannot link two boxes that are geometrically impossible. Wojke and colleagues describe it on arXiv. In the crossing above, two people in different clothing have different embeddings, so the cost matrix is no longer degenerate. Two people in identical uniforms — a factory floor, a football pitch — bring the degeneracy straight back, which is why sports tracking is unusually hard.

Second-pass association on low-score detections. ByteTrack’s observation is that a partially occluded object still produces a detection, just below the confidence threshold, and throwing it away is what creates the gap that becomes a switch. It associates high-confidence detections first, then runs a second association of the remaining tracks against the low-confidence boxes. Zhang and colleagues published it in 2021. It requires no extra network, which is why it displaced heavier methods quickly.

Why MOTA hides association errors

Multiple Object Tracking Accuracy is defined as one minus the sum of false negatives, false positives and identity switches over the number of ground-truth boxes. Put realistic numbers in it:

GT boxes = 12,000   FN = 900   FP = 400   IDSW = 60

MOTA = 1 - (900 + 400 + 60) / 12000 = 1 - 0.1133 = 0.8867

now cut ID switches tenfold, to 6:

MOTA = 1 - (900 + 400 +  6) / 12000 = 1 - 0.1088 = 0.8912

A ten-fold reduction in identity switches — the difference between a usable trajectory dataset and an unusable one — moves MOTA by 0.0045. It cannot be otherwise: switch counts are in the hundreds while detection errors are in the thousands, and the metric adds them. So MOTA is largely a detection metric wearing a tracking name.

Two alternatives fix it. IDF1 scores the F1 of identity assignments over the whole sequence, so a track that is correct throughout scores far above one that is correct in two halves under different numbers. HOTA, from Luiten and colleagues, decomposes explicitly into a detection accuracy and an association accuracy and reports their geometric mean, so neither can be traded away for the other — the HOTA paper is on arXiv. Report HOTA with its two components separately, and report MOTA alongside only because the public leaderboards at MOTChallenge still do.

One more thing the metrics will not tell you: none of them distinguishes a track that was fragmented into three pieces from one that was confidently given the wrong identity, yet the two are very different downstream. Fragmentation loses data and is recoverable by a later re-linking pass over the finished tracks. A confident wrong identity fabricates data — it attributes one person’s movement to another — and nothing downstream can detect it. If your output feeds a per-object statistic, count fragmentations and switches separately and treat the second as the serious one.