Sending an Image to a Model From the Browser

A 12-megapixel phone photo is around 4MB and contains far more detail than any current vision model uses. Resizing it in the browser before it leaves the device makes the upload faster, the request cheaper and the answer no worse. The question is what to resize to, and that has an arithmetic answer.

How much resolution is worth paying for

Vision models do not see pixels. They cut the image into fixed-size patches or tiles, encode each one, and feed the resulting vectors into the model as tokens. The consequence is that image cost is roughly proportional to area, and doubling each side quadruples what you pay.

Tiles for a tile size of 512×512, which is a common choice:

  512  × 512    →  1 × 1  =   1 tile
  1024 × 1024   →  2 × 2  =   4 tiles
  1536 × 1536   →  3 × 3  =   9 tiles
  2048 × 2048   →  4 × 4  =  16 tiles
  4032 × 3024   →  8 × 6  =  48 tiles      (a typical phone photo)

So the phone photo costs about 48× what a single-tile thumbnail costs,
and about 12× what a 1024×1024 version costs.

If a tile is T tokens and there is a fixed base cost B:

  total tokens ≈ B + T × ceil(w / 512) × ceil(h / 512)

The practical conclusion is the one worth acting on. Most vision tasks — describe this, is there a person in it, what does this sign say — saturate well below 1024 pixels on the long edge. Text-heavy images are the genuine exception: reading a dense receipt or a screenshot of a spreadsheet does need resolution, because the model cannot read glyphs that were destroyed by the downscale. So the rule is a rule with one branch, not a constant.

Resizing in the browser

createImageBitmap decodes off the main thread and accepts resize options directly, which makes this both shorter and smoother than the old draw-to-canvas dance. It is available in current browsers; OffscreenCanvas lets the whole thing run in a Web Worker if you are processing several images.

// resize.ts — runs in the browser, no dependencies
export type Resized = { blob: Blob; width: number; height: number; bytes: number };

export async function resizeImage(
  file: File,
  maxEdge = 1024,
  quality = 0.82,
): Promise<Resized> {
  // Decoding a 12MP JPEG is the expensive step; createImageBitmap does it
  // off the main thread and can downscale during decode.
  const probe = await createImageBitmap(file);
  const scale = Math.min(1, maxEdge / Math.max(probe.width, probe.height));
  const width = Math.round(probe.width * scale);
  const height = Math.round(probe.height * scale);
  probe.close();

  const bitmap = await createImageBitmap(file, {
    resizeWidth: width,
    resizeHeight: height,
    resizeQuality: "high",
  });

  const canvas = new OffscreenCanvas(width, height);
  const ctx = canvas.getContext("2d");
  if (!ctx) throw new Error("2d context unavailable");
  ctx.drawImage(bitmap, 0, 0);
  bitmap.close();                       // release the decoded pixels promptly

  const blob = await canvas.convertToBlob({ type: "image/jpeg", quality });
  return { blob, width, height, bytes: blob.size };
}

Three decisions in there are worth stating. JPEG rather than PNG for photographs, because a PNG of a photograph is often five times larger for no visible difference — but PNG for screenshots and diagrams, where JPEG artefacts around text are exactly what destroys the thing you wanted the model to read. Quality 0.82 rather than 0.95, because the difference is invisible after a downscale and the file is roughly half the size. And bitmap.close(), because decoded bitmaps hold real memory and a loop over twenty images without it will make a phone browser reload the tab.

A 4032×3024 phone photo, in practice:

  original JPEG                            ~4,000 KB
  resized to 1024×768, quality 0.82         ~120 KB     ~33× smaller
  resized to 768×576, quality 0.82           ~70 KB     ~57× smaller

Upload time on a 5 Mbit/s uplink, which is a realistic mobile figure:

  4,000 KB × 8 / 5,000 kbit/s  ≈  6.4 s
    120 KB × 8 / 5,000 kbit/s  ≈  0.19 s

That six seconds is time the user spends staring at a progress bar before the model has even started, and it is entirely removable.

Data URL or signed upload

The reason to prefer the third is not elegance. Proxying image bytes through a serverless function means paying for the bandwidth twice, holding the whole file in the function’s memory, and running into request body size limits that are far smaller than people expect. A signed upload removes all three at the cost of one extra round trip.

The signed upload, end to end

1. The browser resizes and asks your server for an upload URL, sending only the content type and byte length.
2. The server authenticates the user, checks quota, validates the declared size and type, and returns a short-lived signed PUT URL plus the object key.
3. The browser PUTs the blob directly to storage. Your server sees none of the bytes.
4. The browser calls your analyse endpoint with the key. The server generates a short-lived signed read URL and sends that to the model.
5. The signed read URL expires in minutes, so the image is never publicly reachable for longer than the request needs.

// app/api/uploads/route.ts — issue the signed URL
import { auth } from "@/lib/auth";

const MAX_BYTES = 8 * 1024 * 1024;
const ALLOWED = new Set(["image/jpeg", "image/png", "image/webp"]);

export async function POST(request: Request) {
  const session = await auth();
  if (!session) return Response.json({ error: "sign in" }, { status: 401 });

  const { contentType, bytes } = await request.json();

  if (!ALLOWED.has(contentType)) {
    return Response.json({ error: "unsupported type" }, { status: 400 });
  }
  // The client's declared size is a hint, not a guarantee. Enforce the real
  // ceiling in the storage policy as well; see the note below.
  if (typeof bytes !== "number" || bytes <= 0 || bytes > MAX_BYTES) {
    return Response.json({ error: "too large" }, { status: 400 });
  }

  const key = "uploads/" + session.accountId + "/" + crypto.randomUUID();
  const url = await signPutUrl(key, contentType, { expiresInSeconds: 120 });

  return Response.json({ url, key });
}

// The browser side, all three steps.
export async function analyseImage(file: File, question: string) {
  const { blob } = await resizeImage(file, 1024);

  // 1. Ask for somewhere to put it.
  const presign = await fetch("/api/uploads", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ contentType: "image/jpeg", bytes: blob.size }),
  });
  if (!presign.ok) throw new Error("upload refused");
  const { url, key } = await presign.json();

  // 2. Send the bytes straight to storage.
  const put = await fetch(url, {
    method: "PUT",
    headers: { "Content-Type": "image/jpeg" },
    body: blob,
  });
  if (!put.ok) throw new Error("upload failed: " + put.status);

  // 3. Ask the server to run the model against the stored object.
  const res = await fetch("/api/analyse", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ key, question }),
  });
  return res.json();
}

Capping the bill

Images make the per-request cost variable in a way text does not, so the caps belong on the server and not in the component that submits the form.

• Cap the pixels server-side, not just client-side. The resize runs in the browser, which means an attacker skips it. Read the stored object’s dimensions before calling the model and reject or downscale anything beyond your ceiling. This is the single most important control on the page.
• Cap images per request and per user per hour. The same limiter as everything else — rate limiting an AI endpoint — but with a cost weight, because one image request is worth many text ones.
• Do not send the same image twice. Hash the resized bytes and cache the answer against the hash plus the question. Users re-ask about the same photo far more often than you would guess.
• Delete the objects. A lifecycle rule that removes uploads after a short window is both a storage-cost control and a privacy control, and it is one line of bucket configuration.

For anything higher volume, the more effective lever is upstream of all of this: a cheap model or a local classifier decides which images are worth sending to an expensive one. Both the cheap multimodal pipeline and image detail levels are about spending the resolution budget where it changes the answer.