What It Costs to Run Object Detection at Scale

There is no published price per image for object detection, and any page that quotes one is quoting a benchmark run on hardware you do not have with a model you are not using. What is portable is the arithmetic. Everything below is derived from inputs named as assumptions in the sentence that uses them.

The cost model

Detection is billed in accelerator-seconds, so the whole model is one division and one multiplication:

gpu_seconds   = images / (throughput * utilisation)
cost          = gpu_seconds * hourly_rate / 3600

cost_per_1000 = 1000 * hourly_rate / (3600 * throughput * utilisation)

Three inputs, and only one of them is a price. throughput is images per second end to end — decode, resize, forward pass, non-maximum suppression, writing the result — measured on your own pipeline, not the figure on a model card, which is almost always forward-pass only at a batch size and precision that may not be yours. utilisation is the fraction of the time you are paying for during which the accelerator is doing that work. hourly_rate is the only number a vendor publishes, and it is the least informative of the three.

The immediate consequence of the third line is that comparing accelerators by hourly rate is meaningless. Cost per image is the rate divided by throughput, so a device costing three and a half times as much per hour is cheaper per image if it is four times faster. That inversion is common enough that the cheap-instance instinct is usually wrong for batch detection work.

A worked figure and its assumptions

Assume a pipeline processing 2,000,000 images a day. Assume you have measured 120 images per second end to end for your model at your input resolution on one accelerator — an assumption standing in for a measurement you must make, not a benchmark reproduced here. Assume an on-demand rate of $1.00 per accelerator-hour, which is a placeholder; check the vendor’s current price page, and see how GPU cloud pricing is structured. Assume, for now, 100% utilisation.

gpu_seconds   = 2,000,000 / 120        = 16,667 s = 4.63 h/day
cost          = 4.63 * $1.00           = $4.63/day
                                       = $139/month at 30 days

cost_per_1000 = 1000 * 1.00 / (3600 * 120) = $0.00231

Now vary one input at a time, which is the part worth keeping:

utilisation 60%   : 4.63 / 0.60      = 7.72 h/day   -> $7.72/day
throughput 60/s   : 2,000,000 / 60   = 9.26 h/day   -> $9.26/day
input 640 -> 1280 : ~4x the FLOPs, so roughly 4x    -> ~$18.5/day
rate $3.50 but 4x faster (480 img/s):
                    2,000,000 / 480  = 1.16 h/day
                    1.16 * $3.50                    -> $4.05/day

The resolution line is the one that surprises people. Convolutional cost scales roughly with pixel count, so going from a 640-pixel input to a 1280-pixel input is about four times the work. Small objects are the usual reason to do it, and “we need to detect things 15 pixels across” is therefore a budget decision as much as an accuracy one. Tiling has the same shape: six tiles per frame is six times the inference.

Decoding is often the actual bill

At 120 images per second the accelerator needs 120 decoded, resized, normalised tensors per second delivered to it. JPEG decoding of a 12-megapixel image is tens of milliseconds of single-threaded CPU work, so one core supplies a small fraction of that rate and the accelerator spends most of its time waiting. Teams discover this as “utilisation is 25% and we do not know why”.

• Count cores as part of the cost. An instance with an accelerator and eight cores may be decode-bound where the same accelerator with thirty-two cores is not, at a higher hourly rate that is still cheaper per image.
• Move decode onto the accelerator. Hardware JPEG and video decode paths exist for exactly this; NVIDIA’s DALI documentation describes building the preprocessing graph so that decode and resize do not run on the CPU at all.
• Decode once. If the same image is scored by three models, decode it once and share the tensor. Pipelines that call three services each doing their own decode pay for it three times.
• Do not decode at full resolution to immediately downscale. JPEG supports DCT-domain scaled decoding at 1/2, 1/4 and 1/8, which is far cheaper than a full decode followed by a resize when the target is a 640-pixel input.

Batch size interacts with all of this. Throughput rises with batch size until the model becomes memory-bandwidth bound, while per-item latency rises roughly linearly, so an offline job should use the largest batch that fits and a real-time endpoint usually cannot. If your workload is offline, that alone is often a two- to three-fold difference in the bill; see when to batch and when to stream.

Utilisation is the multiplier nobody budgets

A reserved instance running around the clock to process a four-hour nightly batch is at roughly 17% utilisation, and you pay for the other 83%. The arithmetic above then understates the real cost by about six times, and no model change will recover it.

• Scale to zero between batches. The cost is cold start: pulling a multi-gigabyte image and loading weights takes long enough that it can dominate a short job. See what a cold start actually costs and how GPU autoscaling behaves.
• Use interruptible capacity for batch. Detection over a queue of images is restartable at the granularity of one image, which is the ideal shape for spot or preemptible instances.
• Check whether you need an accelerator at all. A small detector on CPU through a compiled runtime can be adequate at modest volumes and removes the utilisation problem entirely; see the ONNX Runtime and running inference at the edge.
• Filter before detecting. If most frames contain nothing, a cheap motion or change trigger upstream cuts the image count directly, which is the one term in the model that reduces every downstream cost at once.

The lines that are not compute

Continue the same assumptions: 2,000,000 images a day at 500 KB each is 1 TB a day. At a stated $0.02 per GB-month of object storage, a month of accumulated images is 30 TB, and 30,000 GB at $0.02 is $600 a month — against $139 of compute. Storage overtakes compute in the first month and keeps growing, because compute is a rate and storage is an integral.

compute  : $139 / month   (flat)
storage  : $600 / month after 30 days
           $1,200 after 60 days, $1,800 after 90 ...

Egress is the other line, and the one most often discovered late. Moving images out of a cloud region, whether to on-premises review tooling or to another provider, is charged per gigabyte and is frequently the largest single item in a vision budget. The structural fix is to move the compute to the data rather than the data to the compute, and to send crops and detections rather than frames.

A retention policy is therefore a cost control, not a compliance afterthought. Keeping full frames for the images where the detector was uncertain, thumbnails for the rest, and detections for everything is a reduction of one or two orders of magnitude in the storage integral, and the uncertain frames are also the ones worth relabelling. The general treatment of how these terms compose is in inference economics and cost per request.