Getting Camera and Sensor Data Into an Edge Model Without Falling Behind

A sensor produces data at a rate it chooses. A model consumes it at a rate the hardware permits. When the second is slower than the first — which it usually is — the queue between them is either bounded, or it is a latency bomb with a fuse you can calculate exactly.

Where the frames actually come from

Before designing the buffer, configure the sensor, because several problems that look like queueing problems are actually capture problems.

• Ask the sensor for the resolution you want. Camera hardware supports a discrete set of output configurations, and requesting one close to your model’s input avoids scaling a large frame down on every iteration. Capturing at full sensor resolution to feed a 320-pixel model wastes bandwidth, memory and power on every single frame, and it is the most common avoidable cost in the pipeline.
• Take the native pixel format. Camera pipelines deliver planar YUV-family buffers. Requesting RGB makes something convert every frame; whether that something is efficient hardware or a loop in your own code is worth establishing rather than assuming.
• The camera has its own queue, and it is short. Capture APIs allocate a small fixed number of image buffers and stall or drop when they are all held. Holding onto a delivered frame while you run inference therefore starves capture itself. Copy what you need out of the buffer, release it immediately, and never let a buffer’s lifetime span an inference.
• Frame rate is not constant in low light. Automatic exposure lengthens the exposure time as light falls, and once the exposure exceeds the frame interval the sensor reduces its frame rate to accommodate it. A pipeline validated in a bright office can deliver a fraction of the frames in the evening — with no error and no code change. Cap the exposure duration explicitly if a stable frame rate matters more to you than image brightness, and always log the achieved capture rate rather than the requested one.
• Ask the capture API to discard late frames. Both platforms expose a setting that drops frames rather than queueing them when your callback is slow. Turning it on gives you a latest-value policy for free at the source, and leaving it off is what creates the backlog described next.

The rate mismatch, and what it costs

Put an unbounded queue between a 30 fps camera and a model that manages 20 inferences per second and the arithmetic is unforgiving:

backlog_growth = producer_rate − consumer_rate
               = 30 − 20 = 10 frames per second

staleness(t) = backlog(t) / consumer_rate

  after  10 s:  100 frames queued →   5.0 s behind
  after  60 s:  600 frames queued →  30.0 s behind
  after 600 s: 6000 frames queued → 300.0 s behind

Memory, at 1920×1080 in a 12-bit-per-pixel camera format
(about 3.1 MB per frame):

  after 60 s: 600 × 3.1 MB ≈ 1.9 GB

Both failures are severe and they arrive in a specific order. First the system appears to work but responds to what happened a moment ago; then it responds to what happened a minute ago; then it is killed for memory use. The middle stage is the dangerous one, because a detector reacting to a thirty-second-old frame is not obviously broken, it is just wrong.

There is no configuration of the model that fixes this. As long as the consumer is slower than the producer, the only question is what you throw away and how deliberately.

Dropping frames is the correct behaviour

For nearly all sensor-driven inference, the value of a frame decays to nothing within a few frame intervals. A detection on the current frame is useful; the same detection on a frame from four seconds ago is not just less useful, it is misleading, because the system will act on it as though it were current.

So the correct policy is almost always latest-value-wins: keep the newest item, discard anything older that has not started processing. Explicit exceptions, which are worth naming because they are real:

• Audio. Dropping a window loses part of an utterance and the utterance cannot be reconstructed. Audio pipelines need a genuine ring buffer sized to the worst-case processing delay, and they need the processing to be fast enough to keep up on average.
• Recording or forensic capture. If the data is being stored as evidence, every sample matters and the fix is to decouple storage from inference entirely: write to disk on one path, sample for inference on another.
• Batch processing. A queue of images to be processed when convenient is not a real-time pipeline and should not be designed like one.

The latest-value-wins buffer

The implementation is small, and the important property is that the producer never blocks and never grows anything.

import threading
import time
from dataclasses import dataclass
from typing import Any, Optional


@dataclass
class Sample:
    data: Any
    timestamp: float        # monotonic seconds, taken at capture
    sequence: int


class LatestValue:
    """One slot. The producer overwrites; the consumer takes and clears.
    Counts what it drops, because a silent drop is an unmeasurable one."""

    def __init__(self) -> None:
        self._slot: Optional[Sample] = None
        self._lock = threading.Lock()
        self._new = threading.Condition(self._lock)
        self.produced = 0
        self.dropped = 0

    def put(self, sample: Sample) -> None:
        with self._new:
            if self._slot is not None:
                self.dropped += 1          # overwriting an unconsumed sample
            self._slot = sample
            self.produced += 1
            self._new.notify()

    def take(self, timeout: float = 1.0) -> Optional[Sample]:
        with self._new:
            if self._slot is None:
                self._new.wait(timeout)
            sample, self._slot = self._slot, None
            return sample


def consumer(buf: LatestValue, infer, max_age_s: float = 0.2) -> None:
    while True:
        sample = buf.take()
        if sample is None:
            continue

        age = time.monotonic() - sample.timestamp
        if age > max_age_s:
            # It waited too long behind something else. Acting on it
            # would be worse than skipping it.
            continue

        result = infer(sample.data)
        emit(result, captured_at=sample.timestamp, age_s=age)

Three details carry the design. The producer never blocks, so a slow consumer degrades the frame rate of inference rather than the frame rate of capture. Drops are counted, so “we are processing a third of frames” is a number on a dashboard rather than a suspicion. And the consumer re-checks age after dequeuing, because a sample can become stale while waiting — the slot bounds the queue depth, not the delay.

Timestamp discipline

Once you are dropping data, every downstream consumer needs to know when the surviving data was captured. Four rules:

1. Timestamp at capture, not at processing. The moment the sensor produced the sample, taken as close to the driver as the platform allows. A timestamp applied when inference starts encodes your own queueing delay into what is meant to be a fact about the world.
2. Use a monotonic clock. Wall-clock time can step backwards when the system synchronises, producing negative durations and frames that appear to arrive before their predecessors. Keep a single wall-clock offset if you need absolute time for output.
3. Carry the timestamp through to the result. Every downstream consumer — an overlay, an alert, a log line — should be able to state the age of the data it is showing. A user interface that can display “detected 40 ms ago” can also detect when that number is 4,000 ms.
4. Number the samples. A monotonically increasing sequence number makes drop counting exact rather than inferred, and it makes a reordering bug visible immediately.

Synchronising more than one sensor

As soon as a model takes input from two sensors — two cameras, a camera and an inertial unit, audio and video — you have an alignment problem that dropping makes worse.

• Prefer hardware synchronisation where it exists. If the sensors can share a trigger or a clock, the problem largely disappears and no amount of software cleverness matches it.
• Otherwise, buffer the faster sensor briefly and match on timestamp. Keep a short history of the fast sensor and, when a slow-sensor sample arrives, pick the nearest by capture time. Bound that history in time, not in count.
• Define and enforce a maximum skew. If the nearest match is further away than your tolerance, drop the pair rather than feeding a mismatched one to the model. A model fed misaligned inputs produces confident nonsense.
• Account for per-sensor latency offsets. Different sensors have different pipeline delays between the physical event and the delivered sample. Measure the offset once with a stimulus visible to both — a clap, a flash — and correct for it in the timestamps rather than in the matching logic.

The four numbers to instrument

These four, reported continuously, make every failure in this page diagnosable in one glance:

Report the last one as percentiles rather than a mean, for the reason that applies everywhere else in this library — an average hides the tail, and in a sensor pipeline the tail is the frame on which the system made a decision about something that had already stopped happening.