On-Device Vision: Detection, OCR and Segmentation Within a Frame Budget
10 min read · updated August 4, 2026
On-device vision succeeds or fails on one number: the milliseconds between frames. Everything else — which model, what resolution, whether to run every frame — is a consequence of dividing that budget among the stages that must fit inside it.
Start from the frame interval
frame_interval_ms = 1000 / target_fps 30 fps → 33.3 ms 15 fps → 66.7 ms 10 fps → 100 ms Everything must fit inside one interval: camera callback + format conversion ~3–6 ms resize / normalise into the input tensor ~2–5 ms inference ? post-processing (NMS, decode, sort) ~1–4 ms drawing the overlay ~2–5 ms ---------------------------------------------- leaves for inference at 30 fps: ~15–25 ms
That is the number to take shopping. A model with a 40 ms inference time is not a 30 fps model no matter what its accuracy is, and the honest options are to drop the target frame rate, shrink the input, or decouple detection from display — the third of which is usually correct and is covered below.
The non-inference stages are where surprising amounts of the budget go, and they are much easier to fix than the model. Colour-space conversion from the camera’s native format to RGB, done naively per frame in managed code, can cost more than the model. So can allocating a fresh input buffer each frame.
The tensor arithmetic that decides resolution
Input resolution is the single most expensive parameter in a vision pipeline, and its cost is quadratic. The input tensor alone:
input_bytes = width × height × channels × bytes_per_element 1920 × 1080 × 3 × 4 (fp32) = 24,883,200 B ≈ 24.9 MB 1920 × 1080 × 3 × 1 (int8) = 6,220,800 B ≈ 6.2 MB 640 × 640 × 3 × 4 (fp32) = 4,915,200 B ≈ 4.9 MB 224 × 224 × 3 × 4 (fp32) = 602,112 B ≈ 0.6 MB
And every intermediate feature map scales with the same spatial factor. Going from 224 to 448 on each edge does not double the work — it quadruples the tensor sizes, the memory traffic and, since inference on a device is largely bandwidth-bound, roughly the latency too.
The practical consequence is a rule worth internalising: run the model at the smallest resolution that still resolves the smallest object you must detect, and get resolution back by cropping rather than by upscaling the input. A detector at 320×320 applied to a region of interest often beats the same detector at 640×640 applied to the whole frame, at a quarter of the cost.
Model families and what each costs
Described by cost structure rather than ranked, because the ranking changes and the structure does not.
| Task | Description |
|---|---|
| classification | One label for the whole frame. The cheapest useful vision task: a mobile-oriented backbone at 224×224 fits comfortably in a 30 fps budget on most hardware. Use it as a gate for more expensive stages. |
| object detection | Single-shot detectors — the SSD and YOLO families — predict boxes in one pass and are the only realistic choice on-device. Two-stage detectors are accurate and are not frame-rate models. Cost is dominated by input resolution and by non-maximum suppression on the output. |
| keypoints and landmarks | Face and hand landmark models are small, fast and mature, because they operate on an already-cropped region supplied by a cheap detector. The cascade is what makes them cheap; running them on a full frame is not the intended use. |
| segmentation | Per-pixel output, so the decoder cost scales with output resolution as well as input. Encoder-decoder architectures with a mobile backbone are viable at reduced output resolution — predicting a 128×128 mask and upsampling is standard and usually visually indistinguishable. |
| text detection and recognition | Two models in sequence. See the last section; treating it as one model is the most common mistake here. |
The classical, purpose-built models in this list remain the right answer on-device far more often than people expect. A vision-language model can describe a scene, and it cannot run at 30 fps on a phone. The comparison for document work specifically is set out in vision models versus a real OCR engine; the same reasoning applies with more force when the budget is a frame interval.
Detect rarely, track continuously
The most effective optimisation in on-device vision is not a faster model. It is running the model less often and filling the gaps with something cheap.
- Run the detector every N frames — every fifth frame at 30 fps is detection at 6 Hz, which is ample for objects moving at human speed.
- Between detections, update box positions with a lightweight tracker: optical flow on sparse points, or a simple motion model. This costs a fraction of a detection and keeps the overlay smooth.
- Re-detect immediately when tracking confidence drops, when the scene changes sharply, or when the tracked set has drifted for too many frames.
- Skip work entirely when nothing is happening. A cheap frame-difference check that suppresses inference on a static scene can remove most of the work in real usage, and the energy saving is proportional.
Decoupling also fixes a subtler problem: it lets the display run at the camera’s frame rate while inference runs at whatever rate the device sustains. The alternative — a preview that stutters because it waits for the model — looks broken even when the model is accurate.
Post-processing is where perceived quality lives
A detector emits far more candidate boxes than it has found objects, each with a score. What the user sees is decided by four parameters applied afterwards, and tuning them costs nothing while changing the model costs weeks.
| Parameter | Description |
|---|---|
| score threshold | Below this, candidates are discarded. Set it from a precision–recall curve on your own images, not from the framework default. The default was chosen for a benchmark whose cost balance is not yours. |
| NMS IoU threshold | How much two boxes may overlap before the lower-scoring one is suppressed. Too low and adjacent objects merge into one detection; too high and one object produces three boxes. This is the parameter that produces most 'the model is bad' complaints. |
| per-class vs class-agnostic NMS | Suppressing across classes stops the same object appearing as two labels; suppressing within classes lets a person hold a cup without one erasing the other. Which is right depends entirely on whether your classes physically overlap. |
| minimum box area | Discard detections too small to be actionable. Cheap, and it removes most of the flickering noise at the edge of the detector's range. |
Then add temporal hysteresis, which is the single largest improvement to perceived quality in a live pipeline and which almost nobody implements. A detection appears only after it has been present for two or three consecutive inferences, and disappears only after it has been absent for a few more. Two different thresholds — a higher one to appear, a lower one to persist — stop boxes flickering on and off at the decision boundary. The model has not become more accurate; it has stopped showing the user its uncertainty at 30 Hz, which is what the user was actually objecting to.
Instrument the counts at each stage — candidates in, survivors after thresholding, survivors after suppression, survivors after hysteresis. When detection quality regresses, those four numbers tell you immediately whether the model changed or a threshold did.
The pipeline in practice
- Take the camera’s native format. Cameras deliver YUV-family formats. If your model wants RGB, do the conversion on the GPU or with the platform’s accelerated image APIs, never in a per-pixel loop in application code.
- Pre-allocate and reuse every buffer. The input tensor, the output tensor, the intermediate resize target. Allocating per frame produces garbage-collection pauses that show up as periodic dropped frames and are maddening to diagnose.
- Run inference off the camera callback thread. Block the callback and the camera pipeline stalls, which degrades capture itself rather than just your overlay.
- Drop frames deliberately. If a new frame arrives while inference is running, discard the pending one and keep the newest. Queueing frames turns a throughput problem into an ever-growing latency problem — the reasoning is set out in getting sensor data into an edge model.
- Keep the overlay in the display’s coordinate space. Boxes produced from a rotated, cropped, downscaled tensor must be mapped back through every one of those transforms. Getting this wrong produces boxes that are subtly offset, which reads as a bad model and is not.
OCR is two models, not one
Text extraction on-device is a detection stage that finds text regions and a recognition stage that reads each one. They have completely different cost profiles, and conflating them is why on-device OCR projects miss their budgets.
- Detection runs once per frame on the whole image, at a resolution high enough that small text is not lost. It is the expensive, fixed-cost stage.
- Recognition runs once per detected region, on a small crop. Cheap individually, and the total cost is proportional to how much text is in the frame. A frame containing a page of text can require dozens of recognition passes, so your worst case is nothing like your average case. Cap the number of regions processed per frame and prioritise by area or by proximity to the centre.
Both platforms ship a competent text recogniser in the OS, and using it costs no bundle size and no model maintenance. Reach for a custom model only when you need a script or a domain the platform recogniser handles poorly, and confirm that with a comparison on your own images before committing to shipping and updating a model. The broader pipeline concerns — deskewing, ordering, validating extracted fields — are the same ones described in building an OCR pipeline.