Bounding Boxes and Grounding: Getting Coordinates Out

Plenty of tasks need a location rather than a description: crop the product, redact the signature, highlight the clause, click the button. General vision models will give you coordinates if asked, but they disagree about what a coordinate is, and that disagreement causes more wrong boxes than any weakness in the models themselves.

Three ways to get a box

• Ask the model. No extra infrastructure, works on anything you can describe in words, including relations (“the mug behind the laptop”) that a fixed-vocabulary detector cannot express. Precision is bounded by patch size, and output format is a negotiation.
• Open-vocabulary detectors. OWL-ViT (Minderer et al., 2022) and Grounding DINO (Liu et al., 2023) take a text query and return real detection boxes with real confidence scores. They are purpose-built regression heads, so coordinates are their output rather than a text approximation of one, and they are small enough to self-host.
• Classical detectors. A YOLO-family model on a fixed class list. Fastest and most accurate where your classes are known and you can label training data; useless for anything outside the list.

The useful hybrid, when a general model is doing the reasoning: let the model decide what to find and a detector decide where it is. You keep open-vocabulary flexibility and get coordinates from something that was trained to produce them.

The convention problem

There are at least four live conventions and nothing in the response tells you which one you received:

The failure signature tells you which mistake you made. Boxes that are tiny and clustered in the top-left corner mean you treated normalised values as pixels. Boxes that are transposed — correct shape in the wrong place, mirrored about the diagonal — mean an x/y order mismatch. Boxes that are consistently offset by a constant factor mean a resize you did not account for.

Two rules make this a non-issue. State the convention in the prompt explicitly rather than hoping — asking for “normalised [x0,y0,x1,y1] with the origin at the top left” costs nothing and removes the ambiguity. And draw the boxes on the image during development. Ten seconds of looking beats an hour of reasoning about array order, every time.

IoU, worked

Intersection over union is how box accuracy is measured: the area where predicted and true boxes overlap, divided by the area they cover together. It is 1 for a perfect match and 0 for no overlap.

truth      x0=100 y0=100 x1=300 y1=250     area = 200*150 = 30,000
predicted  x0=120 y0=110 x1=330 y1=240     area = 210*130 = 27,300

intersection
  x: max(100,120)=120 .. min(300,330)=300   -> 180
  y: max(100,110)=110 .. min(250,240)=240   -> 130
  area = 180 * 130 = 23,400

union = 30,000 + 27,300 - 23,400 = 33,900
IoU   = 23,400 / 33,900 = 0.69

0.69 is a box a human would call “basically right” and that the detection literature would score as a miss at the conventional 0.75 threshold. Which is correct depends entirely on what you do next. Cropping a product photo for a thumbnail at 0.69 is fine. Redacting a signature at 0.69 leaves part of the signature visible, which is a data-protection incident rather than an imprecision — so for redaction you dilate every box by a generous margin and accept over-covering.

Choose the threshold from the consequence of being wrong, then measure against it on fifty hand-labelled examples of your own images. Any labelling tool that exports COCO-format JSON gets you ground truth in an hour.

Getting parseable output

Coordinates embedded in prose are a parsing problem you do not need. Where the API supports structured outputs or a JSON schema, use it:

{
  "type": "array",
  "items": {
    "type": "object",
    "required": ["label", "box"],
    "properties": {
      "label": {"type": "string"},
      "box": {"type": "array", "items": {"type": "number"},
              "minItems": 4, "maxItems": 4}
    }
  }
}

Then validate before use: coordinates within range, x0 less than x1, y0 less than y1, and area above some plausible floor. A degenerate box — zero width, or inverted — is a common enough output that handling it belongs in the code rather than in an incident report. And always clamp to the image bounds; a box extending past the edge will throw in some cropping libraries and silently wrap in others.

Multiple instances need explicit handling too, and the schema above quietly assumes you thought about it. Ask for “the box around the signature” on a page with three signatures and you will get one box, chosen arbitrarily, with no indication that others existed. Ask for all of them and you get a list that may contain near-duplicates of the same signature, because nothing deduplicates. The remedy is the detection world’s: run non-maximum suppression over the returned boxes, discarding any box whose IoU with a higher-ranked one exceeds a threshold. Twenty lines of code, and it turns a ragged list into something you can count.

When a detector is the right answer

Reach for a purpose-built detector when you need boxes for many objects per image, when you need confidence scores to threshold on, when latency per image matters, or when volume makes per-request pricing painful — a small open-weight detector runs on modest hardware at many frames per second, which is a different economic regime from a hosted API call per image. Reach for the general model when the target is described rather than classified, when there are only a handful of objects, or when the same call is doing reasoning that a detector cannot do.