The Latency Budget for Analyzing a Livestream in Real Time

“Real time” is not a specification. A useful specification names two endpoints and a number between them, then accounts for every millisecond spent getting from one to the other. What is left over after that accounting is what your model is allowed to take, and it is usually much less than people assume.

What the budget is measured between

A latency budget is meaningless until you say what it is measured between, because the same system has several defensible answers that differ by more than an order of magnitude. Three endpoints matter and they are routinely conflated.

• Glass to glass. Photon hits the camera sensor to the corresponding photon leaving a viewer’s display. This is the number streaming vendors quote and it includes the player’s own buffer, which you do not control.
• Glass to decision. Photon hits the sensor to your system having produced a result about that frame. This is the one an analysis pipeline is actually judged on, and it excludes the viewer path entirely.
• Ingest to decision. Frame arrives at your server to result produced. This is the only span you fully control, and quoting it as though it were the first two is the most common way a latency claim becomes untrue.

Everything below works in glass-to-decision terms, because that is what a moderation trigger, an alert or a live overlay is measured against.

The seven terms in the sum

A glass-to-decision budget is a sum of seven terms. They are worth enumerating in full, because the habit of writing the budget as “network plus inference” hides the two largest ones.

• 1. Capture. Exposure and sensor readout. At the very least one frame interval elapses before a complete frame exists — 33.3 ms at 30 fps, 16.7 ms at 60. Rolling-shutter sensors read out progressively, so the bottom of the frame is genuinely later than the top.
• 2. Encode. The encoder’s own pipeline depth. This is where B-frames live: a codec that reorders frames cannot emit a frame until the frames it references exist, so a reorder depth of two frames is two frame intervals of latency you have chosen to buy. Lookahead rate control adds more of the same. The trade is bitrate, which is the subject of the compression-versus-accuracy page.
• 3. Packetise and upload. Segmenting or packetising, plus the time to push the bytes out of the uplink. On a constrained uplink this scales with segment size and is not negligible.
• 4. Network transit. One-way propagation and queuing to your ingest. Roughly half the round-trip time, plus whatever retransmission the transport does.
• 5. Buffer. The de-jitter or segment buffer at ingest. In segmented protocols this is normally the single largest term in the whole budget and it is a configured value, not a physical one.
• 6. Decode. Turning the compressed bitstream back into frames, including any reorder delay the encoder’s B-frames imposed on the decoder.
• 7. Analysis and action. Preprocessing, the model forward pass, post-processing, and whatever the result triggers — writing a record, dispatching an alert, rendering an overlay. Only the middle of this is what people mean by “inference time”.

Subtracting the fixed parts

With the terms named, the derivation is subtraction. The input figures below are stated assumptions for one plausible configuration — a 30 fps camera, a low-latency segmented ingest, a two-second glass-to-decision target. They are not measurements of any particular system, and the arithmetic rather than the numbers is what transfers.

ASSUMPTIONS (all stated, none measured)
  frame rate                  30 fps      -> 33.3 ms per frame interval
  glass-to-decision target  2000 ms

  1  capture, one frame interval           33 ms
  2  encode pipeline, no B-frames          40 ms
  3  packetise and upload                  60 ms
  4  one-way network transit               80 ms
  5  ingest de-jitter buffer              600 ms
  6  decode                                20 ms
  7b downstream action                    100 ms
                                        -------
     fixed terms                          933 ms

  analysis allowance = 2000 - 933       = 1067 ms per frame

Just over a second, from a two-second target. That is the first useful output of the exercise, and it is already a different conversation from “we have two seconds”. Note where it went: the buffer alone is 600 of the 933 milliseconds of overhead, more than everything else combined. Change the frame rate to 60 fps and the capture term halves to 17 ms, moving the allowance by 16 ms — a rounding error against the buffer. The frame rate matters enormously for the next section and barely at all for this one.

Latency is not throughput

Here is the step that is skipped most often, and it is where real systems fail. An allowance of 1,067 ms per frame does not mean you may take 1,067 ms per frame. At 30 fps a new frame arrives every 33.3 ms. A pipeline that takes a second per frame and processes frames one at a time falls behind by roughly 967 ms for every frame it handles: the queue grows without bound and latency grows with it, so the two-second target is met for the first frame and missed by an arbitrarily large margin an hour later.

The relationship between the two is Little’s Law: for a stable queue, the average number of items in the system equals the arrival rate multiplied by the average time each spends there.

concurrency = arrival rate x latency

  analysing every frame at 30 fps
    30 /s x 1.067 s   = 32 frames in flight
    -> 32 concurrent workers, just to keep up. Latency per frame
       is still 1067 ms; concurrency buys throughput, not latency.

  analysing 2 frames per second (sample 1 in 15)
    2 /s x 1.067 s    = 2.1 frames in flight
    -> 3 workers. Same per-frame latency, 1/15 the hardware.

  the stability condition, either way
    service rate must exceed arrival rate, or delay grows without bound

So the budget produces two separate requirements and they must both be satisfied. The latency requirement says no single frame may take longer than the allowance. The throughput requirement says the aggregate service rate must exceed the sampled arrival rate with headroom. Parallelism satisfies the second and does nothing whatsoever for the first — thirty-two workers each taking a second still take a second. Reducing per-frame latency needs a smaller model, a smaller input resolution, a faster runtime or batching that fits inside the allowance.

The headroom is not optional either. A queue driven at exactly its service rate has unbounded expected delay under any variance in arrival or service time, and both vary in practice: frames are not equal-cost, and traffic is not smooth. Sizing for 70–80% of capacity rather than 100% is what keeps the tail latency near the mean instead of at the timeout.

The term you can actually negotiate

Of the seven terms, capture and transit are physics, encode and decode move by tens of milliseconds, and the action is yours. The buffer is the term with hundreds of milliseconds in it and it is the one set by a protocol choice.

A segmented HTTP protocol cannot deliver a segment before the segment exists, so segment duration is a floor on latency, and players conventionally hold several segments. Apple’s Low-Latency HLS attacks exactly this by publishing partial segments at the live edge so media becomes available before its parent segment closes; the PART-TARGET and PART-HOLD-BACK values in the playlist are where that floor is actually written down for a given deployment, and they are worth reading rather than assuming. A WebRTC ingest avoids segmentation altogether and moves this term into the tens of milliseconds, at the cost of a different operational model.

The general point is that lowering the buffer transfers the cost rather than removing it: a smaller de-jitter buffer means more frames arrive late, and a frame that arrives after its deadline is a frame you either drop or process out of budget. If your analysis tolerates gaps — sampled classification usually does — a small buffer with dropped frames is the right trade. If it does not, because you are tracking objects across frames and a missing frame breaks continuity, the buffer is buying you something real and cutting it is a false economy.