Testing an Image-Plus-Text Prompt Without a Real Image Every Run

A multimodal test that reads a JPEG off disk, base64-encodes it and posts it to a provider is testing four things at once, and only one of them is your code. Three of the four can be fixtured away.

What re-encoding a real image costs you

The obvious approach is to keep a photograph in the repository and encode it in each test. It works, and it degrades in four ways.

• Repository weight. A few megabytes of binary in git, re-fetched by every clone and every CI job forever.
• Runtime. Encoding a large image per test is milliseconds each, which becomes seconds across a suite, and the base64 string is a third larger than the file it came from.
• Non-determinism. If the fixture is fetched from a URL, the test now depends on a host you do not control. If it is re-compressed by any tooling in the pipeline, the bytes change and any assertion on size or hash becomes flaky.
• Cost and licensing. A real photo sent to a provider on every run is billed by image tokens, and a stock image in a public repository is a licence question nobody wants.

The fix is to notice that almost no multimodal test needs the image to depict anything. A test that the payload is well-formed, that the media type matches the bytes, that the size guard fires, or that truncation leaves the image block intact, is indifferent to the picture.

A deterministic fixture

Build the smallest valid image of each format you support, once, and commit the base64 as source. A file of string constants diffs cleanly, costs a few hundred bytes, and never needs decoding tools to review.

// fixtures/images.ts
// A 1x1 transparent PNG, produced once and pasted here. Decodes to 67 bytes.
export const PNG_1x1 =
  "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk" +
  "YPhfDwAChwGA60e6kgAAAABJRU5ErkJggg==";

export const dataUri = (b64, mime) => "data:" + mime + ";base64," + b64;

Where you need something larger — to exercise a size limit, or a tiling threshold — generate it rather than committing it. A few lines with an imaging library producing an N-by-N image of a fixed colour is deterministic, gives a byte-for-byte identical result on every machine for a fixed library version, and can be parameterised by the dimension the test cares about. Assert the generated bytes hash to a known value if you want the determinism itself under test; if that hash changes when the library is upgraded, that is a real thing to know before it changes your payloads.

Keep one genuinely malformed fixture too: a truncated PNG, and a text file with a .png extension. Your upload path should reject both before an encoding step, and that rejection is a code path nobody tests otherwise.

The two payload shapes

The major API families disagree about how an image rides in a message, and if you support both, the shape adapter is the code most worth testing here. In the OpenAI-style chat format, message content is an array of parts and an image is a part carrying an image URL that may itself be a data URI. In the Anthropic-style format, an image is a content block with a source object carrying a type of base64, a media type and the raw base64 data as separate fields — no data URI prefix.

That difference is precisely where the bugs are: passing a full data:image/png;base64, prefix into a field that wants bare base64 produces a decode error from the provider, and stripping it for a field that wants the URI form produces a fetch attempt on a nonsensical URL. Both are one-line adapter bugs, and both are caught by a test that never calls a model.

Assert on the request

Capture the outbound request the same way you would for any wire-format assertion, and check the structure rather than the pixels.

it("sends a bare base64 payload with the declared media type", async () => {
  await describeImage({ png: PNG_1x1, question: "what is this?" });

  const block = captured.body.messages[0].content.find((c) => c.type === "image");

  expect(block.source.type).toBe("base64");
  expect(block.source.media_type).toBe("image/png");
  expect(block.source.data.startsWith("data:")).toBe(false);
  expect(block.source.data).toBe(PNG_1x1);
});

it("puts the question in a text block alongside the image", () => {
  const parts = captured.body.messages[0].content.map((c) => c.type);
  expect(parts).toEqual(["image", "text"]);
});

Four more assertions earn their place. That the encoder ran once per image rather than once per retry — spy on it and assert the call count, because re-encoding inside a retry loop is a real and invisible cost. That the declared media type is derived from the bytes rather than from the filename, tested with the mis-named fixture. That an oversized image is rejected with your own error before the request is built. And that image bytes never reach your logs, which is a one-line assertion on a captured log line and the difference between a debuggable trace and a hundred-kilobyte log entry.

The tests that do need a real image

One category is left, and pretending otherwise is how a suite becomes confident and useless: whether the model reads the image correctly. A 1x1 transparent PNG proves nothing about that, and no amount of payload assertion substitutes for it.

1. Keep a small set — five to fifteen — of real images with known answers, stored as compressed fixtures with the expected values beside them. A receipt with a stated total, a chart with a labelled peak, a form with a specific field filled in.
2. Assert on extracted values, never on the description. The total is 42.50; the highest bar is March; the checkbox is ticked. Those are exact assertions on a non-deterministic output, which is the whole trick of this territory.
3. Run that set nightly or on release rather than per commit. It calls a model, it costs money, and its failures are usually about the model rather than about the commit.
4. Keep the payload tests on the fast path. They are the ones that break when someone refactors the adapter, and they run in milliseconds — the same split argued in testing without the model.