Zero-Shot Object Detection With Open-Vocabulary Models

A closed-vocabulary detector has its class list compiled into its final layer. Open-vocabulary detection replaces that layer’s weights with text embeddings computed at inference, which turns the class list from an architectural constant into an argument. Almost everything else follows from that one substitution.

Replacing the classification head with text

In a conventional detector, the classifier is a matrix of shape (D, C): one learned weight vector of width D per class, fixed when training ended. To score a region you take its D-dimensional feature and dot it against each column.

Open-vocabulary detection observes that the columns do not have to be learned. If the region features live in a space aligned with a text encoder’s output space, you can encode each class name into a vector of the same width, L2-normalise both sides, and use cosine similarity times a learned temperature as the class logit. Adding a class becomes encoding a string.

closed vocabulary:   logits = region_feat @ W          W is (D, 80), learned
open vocabulary:     logits = norm(region_feat) @ norm(text_emb).T * temp
                                                    text_emb is (D, K),
                                                    K = however many strings
                                                        you passed in

The box head is untouched. Localisation was never class-specific in the first place — a detector trained on eighty classes learns a general objectness and box regression that transfers to objects outside them — and that asymmetry is the reason the trick works at all. The hard part of detection is what is where; the easy part, in this framing, is naming it.

Two lineages: similarity and grounding

Similarity-based. ViLD distils CLIP’s image embeddings into a detector’s region embeddings so the regions land in CLIP space. OWL-ViT — Minderer and colleagues, “Simple Open-Vocabulary Object Detection with Vision Transformers” (ECCV 2022) — goes further and simplifies: take a contrastively pretrained vision transformer, remove the final pooling, and attach a box head and a class-embedding head to each output token, so every patch token becomes a candidate detection. OWLv2 scales the same recipe with self-training on pseudo-annotated web image-text pairs. A property of this family that gets overlooked: because the class side is just an embedding, you can supply an image crop instead of a string as the query, which is the practical answer for an object your vocabulary has no word for.

Grounding-based. GLIP reformulates detection as phrase grounding: the text is a caption, and the model predicts, for each box, alignment scores over the caption’s tokens. Grounding DINO extends this with fusion at several depths — a cross-modality feature enhancer, language-guided query selection, and a cross-modality decoder — rather than a single dot product at the end, which is why it handles multi-word and referring expressions better. Its prompt convention is a list of categories separated by full stops, as in “person . forklift . pallet .”; separating the phrases matters, because encoding all the class names as one running sentence lets attention bleed between concepts.

Neither is magic, and it is worth being precise about where the ability comes from. It comes from large image-text pretraining, so the vocabulary a model can detect is bounded by what its text encoder learned. “Excavator” and “anterior cruciate ligament” are in web-scale corpora; your internal part number is not, and no prompt phrasing will make it so. Benchmarks report novel class performance on the LVIS rare split, which is a claim about label supervision rather than about concept exposure — the concept may well have been in the pretraining captions.

A worked query

from transformers import OwlViTProcessor, OwlViTForObjectDetection
from PIL import Image
import torch

processor = OwlViTProcessor.from_pretrained("google/owlvit-base-patch32")
model = OwlViTForObjectDetection.from_pretrained("google/owlvit-base-patch32")

image = Image.open("warehouse_aisle.jpg")
queries = [["a photo of a forklift",
            "a photo of a pallet",
            "a photo of a person in a hi-vis vest"]]

inputs = processor(text=queries, images=image, return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)

# logits    (1, num_patches, num_queries)
# pred_boxes(1, num_patches, 4)  cx, cy, w, h normalised to the input

results = processor.post_process_object_detection(
    outputs, threshold=0.1,
    target_sizes=torch.tensor([image.size[::-1]]))

# a person in a hi-vis vest  (418, 260, 512, 604)  0.34
# a pallet                   (712, 540, 928, 661)  0.21
# a forklift                 (640, 300, 1002, 668) 0.18

Two details in that snippet are load-bearing. The boxes come back as normalised centre-x, centre-y, width, height relative to the model’s square input, so they need the letterbox inverse applied before they mean anything in the original frame. And the query strings are wrapped in the CLIP prompt template — bare class names typically score lower than “a photo of a ...”, because the template matches the caption distribution the text encoder was trained on.

The class name is a hyperparameter, and treating it as one is the single highest-return practice with these models. “Hi-vis vest”, “safety vest” and “reflective jacket” retrieve different things. Write four or five candidate phrasings, evaluate them on fifty labelled images, keep the best. Fifty images is not enough to train anything and is plenty to choose a string.

Why one threshold does not work

This is the failure mode that surprises teams in deployment. The score for a box is a cosine similarity against a particular text vector, and there is no mechanism anywhere in training that puts the score distributions for different text vectors on a common scale. The similarity between region features and “pallet” may concentrate around 0.2 while “person” concentrates around 0.35, entirely because of how those words sit in the embedding space. A single global threshold therefore over-detects one class and under-detects another, and no amount of tuning finds a value that is right for both.

• Calibrate per query. Set a separate threshold for each phrase on a small labelled set. This is the only approach that reliably works, and it is cheap.
• Or rank instead of thresholding. If you know roughly how many instances to expect, take the top k per query per image and drop the absolute scale entirely.
• Add negative queries. Including phrases for what is not of interest — “a photo of an empty floor”, “a photo of a cardboard box” — gives the comparison somewhere else to put mass and sharpens the margin.
• Suppress across queries. Different phrases will fire on the same object. Run NMS across all queries, not within each, or every forklift is also a pallet.

Notice what the first bullet implies: once you have a labelled set large enough to calibrate thresholds, you are no longer in a zero-shot setting, and a few-shot or fine-tuned closed-vocabulary detector on the same labels will usually be both more accurate and far faster. That is not an argument against these models — it is a description of where their value actually is. See few-shot image classification for the adjacent regime.

What it still cannot do

• Relations and composition. “The cup to the left of the laptop” mostly fails, because each box is scored independently against the text and nothing in the scoring represents a relation between two boxes. Grounding-based models do better on referring expressions than similarity-based ones, but not reliably.
• Attribute binding. “A red car and a blue truck” leaks: the colour attaches to the scene rather than to the specific object, a known weakness inherited from contrastive image-text pretraining.
• Negation. “A shelf without a price tag” is not representable. There is no vector for the absence of a thing.
• Fine-grained distinctions. “A 10 mm hex bolt” requires a distinction the text encoder cannot make and the image resolution probably cannot support.
• Throughput. A transformer-based open-vocabulary detector is typically an order of magnitude heavier than a small single-stage detector, which puts it outside the budget for continuous video; see what detection costs at scale.

The workflow those limits point at is the one most teams converge on: use the open-vocabulary model to bootstrap. Run it over unlabelled data, correct its output rather than annotating from scratch, and train a small closed-vocabulary detector on the result to run in production. The zero-shot model does the expensive part — producing candidate labels — once, and never has to meet the latency budget.