Image Annotation Formats: COCO, YOLO and Pascal VOC Explained

The three formats disagree about the origin, the units, whether the image size is stored, and where class indices start. Every one of those disagreements produces a dataset that trains without error and gives worse results than it should.

Three formats, three files on disk

Pascal VOC is one XML file per image. It carries an image size element with width, height and depth, and one object element per annotation containing a class name as a string, a difficult flag, and a bndbox with xmin, ymin, xmax, ymax in absolute pixels. Class names are strings, so there is no index to get wrong.

COCO is a single JSON file for the whole split, with three top-level arrays: images, annotations and categories. An annotation references an image by image_id and a class by category_id, and stores the box as a bbox array of four numbers in absolute pixels, ordered [x, y, width, height] with the origin at the top-left corner of the box — not the centre, and not a second corner. This is the most common misreading of the three formats.

YOLO is one plain-text file per image, one annotation per line: a zero-based integer class index followed by four numbers normalised to the range 0–1, ordered x_centre y_centre width height. The class names live in a separate list file whose order defines the indices. Nothing in the annotation records the image dimensions, which is the single most consequential thing about this format.

One box, converted three ways

One car in a 1280×720 frame, spanning x from 384 to 640 and y from 210 to 498:

image: 1280 x 720

Pascal VOC   xmin 384   ymin 210   xmax 640   ymax 498

COCO         x = xmin                = 384
             y = ymin                = 210
             w = xmax - xmin = 640-384 = 256
             h = ymax - ymin = 498-210 = 288
             bbox = [384, 210, 256, 288]   area = 73728

YOLO         xc = (384 + 640) / 2 = 512   -> 512 / 1280 = 0.400000
             yc = (210 + 498) / 2 = 354   -> 354 /  720 = 0.491667
             w  = 256                     -> 256 / 1280 = 0.200000
             h  = 288                     -> 288 /  720 = 0.400000
             line: 0 0.400000 0.491667 0.200000 0.400000

back to VOC  xc*W = 512   w*W = 256   ->  xmin = 512 - 128 = 384
             yc*H = 354   h*H = 288   ->  ymin = 354 - 144 = 210

The round trip is exact here because the numbers divide cleanly. It usually does not: 0.491667 is a rounded value, and writing six decimal places instead of the full float costs up to half a pixel on this image and proportionally more on a larger one. That is irrelevant for training and matters if you are round-tripping a dataset repeatedly through a conversion script, because the error accumulates. Keep one format as the source of truth and regenerate the others, rather than converting a converted set.

The normalisation is also what makes YOLO labels resolution-independent — resize the image and the labels remain correct, which is convenient. The flip side is that a YOLO label file is meaningless without knowing the image it belongs to, so a mismatched or missing image silently produces a box in the wrong place rather than an error. Any converter you write should read the dimensions from the image file itself, never from an assumption.

The off-by-one that survives every round trip

The conversion above used w = xmax - xmin. Whether that is right depends on something the format does not state: is xmax the last pixel inside the box, or the first pixel outside it? If it is inclusive, the width is xmax - xmin + 1 = 257, not 256.

The original Pascal VOC ground-truth XML files use one-based pixel coordinates, which is why detection codebases that read VOC frequently subtract 1 on load; most modern annotation tools that export “Pascal VOC XML” write zero-based coordinates instead. Both files look identical. Do not resolve this by trusting the format name — resolve it by measuring your own data:

# over every annotation in the set
min(xmin) == 0  -> zero-based, exclusive xmax is the safe assumption
min(xmin) == 1  -> one-based; check whether max(xmax) == image_width
                   (inclusive) or image_width + 1 (which means something
                   upstream already converted badly)
max(xmax) > image_width  -> boxes are out of bounds; clip before training

A one-pixel error is invisible on a 288-pixel-tall car and is not invisible on a 12-pixel object, where it is 8% of the box. It also shifts every IoU slightly, so a validation score computed with one convention against predictions made under the other is quietly and consistently pessimistic. Check it once per dataset and record the answer next to the data.

Category ids: the conversion bug that shifts every label

This is the one that produces a model that trains fine and predicts the wrong class for everything. COCO’s category_id values are one-based and, in the standard COCO detection set, not contiguous: 80 classes carry ids running up to 90, with gaps left by categories that were removed after the ids were assigned. YOLO’s class indices are zero-based and must be contiguous from 0 to n−1.

So the conversion is not yolo_id = coco_id - 1, however much it looks like it should be. It is a lookup: sort the categories, build an explicit map from category id to a dense index, and write that map next to the dataset. Getting it wrong by the naive subtraction shifts every class above the first gap by one, so a model trained on it learns confidently mislabelled data and reports a respectable loss curve throughout. The symptom is a confusion matrix whose mass sits one off-diagonal, and if you are not looking at a confusion matrix you will not see it at all.

Pascal VOC dodges the problem by storing class names as strings, at the cost that a typo — motorbike against motorbike with a trailing space — creates a silent extra class. Counting distinct class names across the whole annotation set before conversion takes one command and catches it.

Beyond boxes: masks, keypoints and crowds

The three formats diverge much further once the task is not detection.

• Segmentation. COCO stores a mask either as a list of polygon vertex coordinates or as run-length encoding for complex shapes, in the annotation’s segmentation field. Pascal VOC stores masks as separate paletted PNG files, with a distinguished void value marking the ambiguous border pixels that evaluation ignores. YOLO’s segmentation variant extends its text line with normalised polygon coordinates. A polygon-to-RLE conversion is lossy in one direction and not the other, so the choice of source format constrains what you can produce later.
• Keypoints. COCO stores a flat array of x, y and a visibility flag per joint, with 0 for unlabelled, 1 for labelled but occluded and 2 for labelled and visible — and the ordering of the 17 joints is part of the specification, as covered in pose estimation. A converter that drops the flag turns every occluded joint into a visible one at coordinates the annotator guessed.
• Crowds and difficulty. COCO marks a group of indistinguishable instances with iscrowd set to 1, and its evaluation treats those regions as ignore areas rather than as negatives. Pascal VOC serves the same purpose with its difficult flag. A converter that discards either turns a region the benchmark deliberately excludes into ordinary training signal, which teaches the model something the metric will punish it for.