Mapping Image and File Inputs Between Chat APIs

Every chat API that accepts images does the same thing: a turn’s content becomes a list of parts and one of the parts is an image. The part is named differently, the bytes are carried differently, and one API has a resolution control the others have no field for.

The shared concept

Before the differences, the thing they agree on, because it is what makes a portable adapter possible at all. In a text-only request, a turn’s content is a string. Once images are involved, the content becomes an ordered array of typed parts — some text, some image — and the order is meaningful. “Here is a chart, what is the trend?” and “What is the trend? Here is a chart” are different inputs, and the conventional ordering is to put the image before the question about it.

They also agree that an image belongs in a user turn. None of these APIs lets you put an image in an assistant turn as though the model had produced it, which matters when you replay a history containing generated images — there is nowhere to put them, and they must be re-attributed to a user turn with an explanatory text part or dropped.

One image, three shapes

The same photograph, base64-encoded, attached to the same question.

// OpenAI Chat Completions — content parts, image carried by a data URL
{ "role": "user", "content": [
    { "type": "text", "text": "What is on this receipt?" },
    { "type": "image_url",
      "image_url": { "url": "data:image/jpeg;base64,/9j/4AAQSkZJRg...",
                     "detail": "high" } }
] }

// Anthropic Messages — content blocks, media type separate from the bytes
{ "role": "user", "content": [
    { "type": "text", "text": "What is on this receipt?" },
    { "type": "image",
      "source": { "type": "base64",
                  "media_type": "image/jpeg",
                  "data": "/9j/4AAQSkZJRg..." } }
] }

// Google Gemini generateContent — parts, inline data
{ "role": "user", "parts": [
    { "text": "What is on this receipt?" },
    { "inline_data": { "mime_type": "image/jpeg",
                       "data": "/9j/4AAQSkZJRg..." } }
] }

Read those three carefully and the conversion rules fall out. The bytes are the same base64 in all three. What differs is whether the media type is a separate field or embedded in a data URL prefix — which is the single most common conversion bug, because splitting a data URL on the comma and forgetting to parse the type before it produces a request that is rejected with an unhelpful message about an unsupported image, or worse, accepted with the wrong type declared.

The other difference visible above is the detail field on the OpenAI shape, which controls whether the image is processed at low or high resolution and therefore how many tokens it costs. There is no counterpart on the other two. Converting away from it, the field simply disappears, and the behaviour you get is whatever the target does by default.

Why base64 is the portable floor

Every one of these APIs also accepts, or has accepted, a way to reference an image without inlining it: an https URL the provider fetches, or a handle from a file-upload endpoint. Those are better in every respect except portability — smaller requests, no re-upload of the same image across turns, no encoding cost — and they are precisely the part that differs most between providers and changes most often.

So an adapter that must work across providers converges on base64 as its floor, and pays for it. The costs are worth stating plainly because they decide whether an integration works at scale:

• Payload inflation of roughly a third. Base64 encodes three bytes as four characters, so a 3 MB photograph becomes about 4 MB of JSON. Providers cap request body size, and a multi-image request is a common way to exceed it. The failure is a 413 or a generic request-too-large error that names no image.
• Re-sending on every turn. An image in turn three is in the history for turns four, five and six. A ten-turn conversation with one image sends that image ten times, both in bytes and, unless prompt caching applies, in tokens.
• Fetch failures move to your side. With a URL, a broken link is the provider’s fetch failing. With base64, you are the one fetching and encoding, which is better for reliability and means your timeouts and retries now cover image retrieval too.

The practical consequence is that resizing before encoding is not an optimisation, it is part of the integration. Sending a 12-megapixel phone photo to a model that will downscale it anyway costs you request size, latency and tokens for nothing. Resize to the largest dimension the model usefully reads, strip metadata, and re-encode as JPEG unless transparency matters.

What does not survive

• The resolution control. Discussed above. Converting towards an API without one, record that the request asked for low detail and could not get it, because the token cost of the request will not match what the caller expected.
• URL references. Where the target does not fetch URLs, your adapter must fetch and encode — which means it now makes outbound requests to whatever URL a caller supplied. That is a server-side request forgery surface, and it needs an allowlist, a size cap and a timeout before it ships.
• Accepted media types. The intersection is narrower than any single provider’s list. JPEG and PNG are safe everywhere; WebP, GIF and various less common formats are not universally accepted, and animated formats are handled inconsistently where they are. Normalise to JPEG or PNG at the adapter boundary rather than discovering the gap per provider.
• Images per request. Providers cap the number of images in one request and the caps differ. A request built for a generous limit fails wholesale on a stricter one, so the cap belongs in the capability table alongside the rest.
• Token accounting. How an image is converted into tokens is provider-specific and dimension-dependent, so the same image costs different amounts on different providers and there is no conversion factor. Do not estimate image tokens from a formula you found; read the usage object on the response, which is the only number that matches your bill. This library’s token count mismatch page covers why local counts and billed counts diverge generally, and images are the sharpest instance of it.

Files that are not images

PDFs and other documents are a separate capability from images and it is worth not conflating them. Some APIs accept a PDF as a distinct part type with its own carrier — a document block with a base64 source and a media type, or a file handle from an upload endpoint — and process it natively, including the page images. Others accept no such part, and the only route is to do the conversion yourself: extract the text, or render each page to an image and send the pages as images.

Those two paths give measurably different results and the difference is not subtle. Extracted text loses layout, tables and anything in a figure. Page images preserve all of it and cost image tokens per page, which for a forty-page document is a large request. Where a provider supports native document input, it is usually doing something closer to the second. When you port to a provider that does not, decide deliberately which approximation you are making rather than defaulting to text extraction because it is the easier code — if your prompt asks about a table, text extraction will quietly stop working.

The adapter that holds up

One internal representation, normalised early, serialised per provider. The representation that survives contact with all three shapes is: decoded bytes or a fetchable reference, a media type as a separate field, the pixel dimensions, and an optional detail preference that the serialiser may drop.

type ImagePart = {
  kind: "image";
  bytes: Uint8Array;        // always decoded internally, never a data URL
  mediaType: "image/jpeg" | "image/png";
  width: number;
  height: number;
  detail?: "low" | "high";  // honoured where supported, recorded where dropped
};

Holding decoded bytes rather than a data URL is the decision that saves the most trouble. A data URL couples the media type to the payload string, and every serialiser then has to parse it back apart — which is exactly the step people get wrong. Encode at the boundary, once per provider, from a representation where the type is already a separate field.

Normalise on the way in: fetch if it is a reference, decode, check the real media type from the bytes rather than trusting a supplied content-type header, resize if either dimension exceeds your cap, re-encode to JPEG or PNG. Then the per-provider serialiser is a dozen lines each and the awkward parts — the SSRF allowlist, the size cap, the format normalisation — exist in one place instead of three.