Build an Alt-Text and Captioning Service

Most automated alt text fails not because the description is wrong but because it is the wrong kind of description. WCAG 2.2’s success criterion 1.1.1 (Non-text Content, Level A) requires a text alternative that serves the equivalent purpose — and the purpose of a decorative flourish, a button icon, a photograph and a chart are four different purposes. Route them separately and the output becomes usable.

Four kinds of image, four different answers

A generator that produces “a chart showing a line going up” has technically described the image and conveyed none of its content. That is the standard failure and the class routing is the whole fix.

Context is most of the answer

The same photograph needs different alt text on a news article and in a camera review. A model that sees only the pixels cannot know which. So the pipeline’s job is largely to gather context and hand it over.

• Surrounding text — the paragraph before and after, and the nearest heading. This alone changes the output more than any prompt wording.
• Existing metadata — the filename, an existing caption, a title attribute, the link target if the image is inside an anchor. A filename like ceo-margaret-tan-2024.jpg contains a fact no vision model will recover.
• Position and size — a 24×24 image inside a button is functional; a 40×40 spacer is decorative; a 1200-pixel-wide image under a heading is informative. Rules on dimensions and DOM position classify a large fraction of a real site before a model is called at all.
• Repetition — the same image URL appearing on two hundred pages is a logo or an icon. Group by hash first and describe each distinct image once.

The prompts

CLASSIFY = """Classify this image's ROLE on the page, using the image and
the context. Reply with one word:

decorative  - conveys nothing the reader needs; purely visual
functional  - it is a control or a link; its meaning is what it does
informative - it carries content the surrounding text does not
complex     - a chart, diagram, map or table image with data in it

If a sighted reader would lose nothing if the image vanished, it is
decorative."""

INFORMATIVE = """Write alt text for this image.

- One sentence, under 125 characters, ending in a full stop.
- Describe what a sighted reader would take from it IN THIS CONTEXT.
- Do not begin with "image of", "photo of" or "a picture showing".
- Do not repeat information already in the caption or nearby text.
- Name people only if the context names them. Never guess a name.
- Do not describe race, age, weight or presumed gender of people. Describe
  what they are doing.
- If the image is too ambiguous to describe usefully, reply exactly:
  NEEDS_HUMAN"""

COMPLEX = """This is a chart or diagram. Produce JSON:
{"alt": "under 125 chars: what kind of chart and what it is about",
 "long": "the content: axes, units, series, the trend, and any value
          explicitly labelled. Do not estimate unlabelled values.",
 "data_visible": true|false}
Set data_visible false if the values are not legible; then 'long' must say
the chart's values could not be read."""

The instruction not to estimate unlabelled chart values is the one that prevents the worst outcome here. A model asked to describe a chart will read approximate values off the axis and state them as facts, and a screen reader user has no way to know the number was inferred. Refusing is correct.

The 125-character guidance is a convention rather than a rule in the specification — several screen readers historically truncated long alt attributes, and short alternatives are easier to listen to. Treat it as a target, and put anything longer in a proper long description.

The batch pipeline

1. Inventory. Crawl or query for every image with its URL, dimensions, existing alt, surrounding text and page URL. Store it. This step alone produces the report of how many images have no alt at all, which is usually the number that gets the work funded.
2. Hash and group. SHA-256 of the image bytes. Describe each distinct image once and fan the result out to every occurrence — but keep the per-occurrence context, because the same image can be decorative in one place and informative in another.
3. Rule pass. Classify by dimensions, file name and DOM position where you can. Tiny images, images inside buttons with a visible text label, and repeated background images do not need a model.
4. Model pass. Classify, then generate with the prompt for that class. Two calls per image, and the first one is cheap because it needs only a low-detail image.
5. Validate. Length, no forbidden openings, not identical to the caption, not identical to the file name, not empty for an informative image. Reject and queue rather than publish.
6. Write back with a diff. Never overwrite existing alt text automatically. A human-written alternative is more likely to be right than a generated one, and replacing it is a regression that nobody will notice for a year.

Where a provider offers a batch endpoint at a reduced rate for non-urgent work, this is the ideal workload for it — nothing here is interactive. Batch endpoints typically trade latency for a discount, and a catalogue re-description that finishes overnight is exactly the trade you want.

The review lane

Three things go to a human, and the first is the one people leave out:

• Anything the model marked NEEDS_HUMAN or where data_visible is false.
• Every image classified decorative. The cost of a false decorative is total — the information is silently dropped for every screen reader user and nothing in the page indicates it. False positives in the other direction merely produce noise.
• A random 2 per cent of everything else, forever. This is your quality measurement, and without it you cannot tell that a model change degraded the output.

Show the reviewer the image, the context and the generated text together, with keyboard-only accept and edit. If reviewing is slower than writing the alt text from scratch, the lane will be abandoned and the generated text published unchecked.

What a catalogue costs

50,000 images on a site, of which 30% are duplicates by hash
  = 35,000 distinct images to describe

Per image:
  classify   low-detail image + ~120 tokens of context + ~5 tokens out
  describe   standard-detail image + ~250 tokens of context + ~60 tokens out

Image tokens are the term that varies MOST between providers: the same
photograph can be a few hundred tokens at low detail and a few thousand at
high detail, computed from its dimensions by a published formula. Look up
that formula for your provider and put a real number in, because it, not
your prompt, sets the bill.

Text-side arithmetic you can do now:
  35,000 x (120 + 250) input text tokens  = 13.0M tokens
  35,000 x (5 + 60)   output tokens       =  2.3M tokens

The lesson: at 35,000 images the fixed cost is dominated by image tokens,
so the leverage is (a) de-duplicate hard, (b) rule out decorative images
before any call, (c) use low detail for the classification pass.

Testing alt text without a screen reader

You should test with a screen reader eventually, and with someone who uses one daily if you possibly can. But three cheap tests catch most defects long before that, and they are automatable enough to run over a sample of every batch.

1. The images-off test. Render the page with images replaced by their alt text. Read it. If a sentence now says “photograph of a smiling woman” where the caption already said who she was, the alt is redundant; if a paragraph now makes no sense, the alt is missing information. This finds duplication and gaps in one pass and needs no assistive technology at all.
2. The description-to-image test. Give the generated alt text — without the image — to a person or a second model and ask them to pick the matching image from a set of five from the same page. Alt text that does not identify its own image among near neighbours is too generic to be useful, and this is a measurable pass rate you can track across model changes.
3. The mechanical checks. Alt identical to the file name, alt identical to the caption, alt beginning with “image” or “photo”, alt over 200 characters, alt on a decorative image, missing alt on an informative one, and the same alt repeated on every image of a page. Each is a few lines and each corresponds to a real defect people ship.

Track the second test’s pass rate as your quality number rather than any judgement score. It is objective, it is comparable across model versions, and it fails in the right direction: text that is vaguely true of many images scores badly, which is precisely the failure automated alt text has.

Where automated alt text should not be used

• Images of text. The alternative must be the text, exactly. OCR it; do not describe it. A model paraphrasing a sign or a slide loses the content it was supposed to carry.
• Medical, legal or safety imagery. A wrong description here is not an accessibility issue, it is a harm. Human authorship only.
• Identifying people. Unless the surrounding text names them, a description that names or characterises individuals is a fabrication with a privacy dimension attached. Vision models assert details that are not in the image, and faces are where they do it most confidently.
• As a compliance claim. Generated alt text moves a site towards the intent of accessible content; it does not by itself establish conformance, and an overlay that claims otherwise is the well-documented anti-pattern of this field.