Migrating Multi-Modal Prompts Between Providers

An image in a prompt is not one field, it is four decisions — how the bytes are encoded, where the media type is declared, whether a URL may be used instead, and how much the provider will downscale before the model sees it. Providers differ on all four, and the first two produce failures that look like corrupt image data.

The two content-block shapes

Both major APIs put images in the same structural place: the content of a user message becomes an array of parts rather than a string, and one of those parts is an image. What differs is the shape of that part.

Anthropic’s Messages API uses a part of type image with a nested source object. The source discriminates on its own type — base64, carrying separate media_type and data fields, or url, carrying a url. The media type is a field of its own.

// Anthropic: media type is a separate field, data is raw base64.
{
  "role": "user",
  "content": [
    { "type": "image",
      "source": {
        "type": "base64",
        "media_type": "image/jpeg",
        "data": "/9j/4AAQSkZJRgABAQAA..."
      } },
    { "type": "text", "text": "What is wrong with this chart?" }
  ]
}

OpenAI’s Chat Completions API uses a part of type image_url whose image_url object carries a single url string. That string is either an ordinary HTTP URL or a data URI — and in the data-URI case the media type is encoded inside the string rather than declared beside it. Its Responses API uses a different part type again, input_image, with the URL as a direct string field, so a migration between two surfaces of the same vendor is also a rewrite.

// OpenAI Chat Completions: media type lives inside the URL string.
{
  "role": "user",
  "content": [
    { "type": "image_url",
      "image_url": {
        "url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAA...",
        "detail": "high"
      } },
    { "type": "text", "text": "What is wrong with this chart?" }
  ]
}

The data-URI prefix problem

This is the failure worth internalising, because it is the one that wastes an afternoon.

The base64 payload is identical between the two. What differs is whether it is wrapped in a data URI. Moving in one direction, a developer takes the working url string and drops it whole into the data field — so data now begins data:image/jpeg;base64,, which is not valid base64, and the request fails with a message about an unreadable image. Moving in the other direction, the raw base64 is placed in url with no prefix, and the provider cannot tell it is an image at all.

Both failures read as “my image is broken”, and both send people off to re-encode a file that was fine. The translation is one line each way, and it belongs inside your provider implementation rather than at the call site:

const DATA_URI = /^data:([^;,]+);base64,(.*)$/s;

/** Data URI -> { mediaType, data } for a separate-field API. */
export function splitDataUri(uri: string) {
  const m = DATA_URI.exec(uri);
  if (!m) throw new Error("not a base64 data URI");
  return { mediaType: m[1], data: m[2] };
}

/** { mediaType, data } -> data URI for a single-string API. */
export function toDataUri(mediaType: string, data: string) {
  return `data:${mediaType};base64,${data}`;
}

Two adjacent traps. Base64 with embedded newlines — the output of a command-line encoder that wraps at 76 columns — is rejected by at least one API, so strip whitespace before sending. And the declared media type must match the actual bytes: a PNG file declared as image/jpeg is rejected even though the base64 is perfectly valid, which reads like an encoding fault and is not one.

Size, resolution and count limits

Three separate limits apply, and they are commonly confused with one another.

• Per-image byte size. A cap on the encoded payload. Remember base64 inflates bytes by about a third, so a file comfortably under the limit on disk can exceed it on the wire.
• Total request size. Separate from the per-image cap, and the one that bites on multi-image prompts. Five images each under the per-image limit can still exceed the request limit together, and the resulting error names the request, not the image.
• Pixel dimensions. Providers downscale images above a long-edge threshold before the model sees them. Sending a 4000-pixel screenshot to an API that resizes at a lower threshold costs upload bandwidth for detail that is discarded. Worse, if you are relying on fine text in the image being legible, the downscale can silently destroy it — the request succeeds and the answer is simply wrong.

Image token cost follows from the post-resize dimensions, which is why two providers can charge noticeably different amounts for the same file. The resize threshold is the number to look up, not the byte cap, if you care about cost or about legibility of fine detail.

The fields with no counterpart

These are the lossy parts, and an adapter has to make a decision about each rather than silently dropping it.

• A detail or fidelity hint. OpenAI’s detail field lets you trade resolution for token cost explicitly. There is no equivalent parameter on APIs that resize by fixed policy, so the only way to reproduce a low-detail request is to downscale the image yourself before encoding. That is a real behavioural difference: code that relied on detail: "low" to keep costs down will silently get full-resolution billing after a move unless you add client-side resizing.
• URL fetching. Where a provider accepts a URL, the provider fetches it — which means the URL must be publicly reachable from their network, not from yours. A signed URL with a short expiry, or an internal host, works in development and fails in production. Base64 has no such dependency and is the portable choice.
• File references. Both ecosystems offer an upload endpoint returning an id you reference instead of inlining bytes, which is the right answer for an image used across many requests. The ids are not portable and the reference block shape differs, so an adapter that supports file ids needs a per-provider upload path as well as a per-provider reference format.

Rewriting the prompt

1. Represent the image in your own code as { mediaType, bytes } — never as a provider-shaped block and never as a pre-built data URI. Every conversion then happens at the edge.
2. Encode at the edge, in the provider implementation: build a data URI for APIs that want one, and emit separate fields for APIs that want those. Strip whitespace from the base64 in both paths.
3. Resize before encoding, to the smaller of the two providers’ thresholds. You lose nothing the model would have seen and you cut upload time and image tokens on both.
4. Order the parts deliberately. Put the image before the text that asks about it; several providers document this ordering as producing better results, and it costs nothing to be consistent.
5. Send one real image through both paths and compare the answers on content you can verify — a chart with a number in it, a screenshot with small text. A resize difference shows up as a wrong number, not as an error.
6. Add a size check before the call that fails loudly with your own message when an image exceeds the target provider’s cap. A clear local error beats a 400 whose text you have to look up.