Running a Model in the Browser With WebGPU

WebGPU gives a web page access to the GPU, which makes running a language model inside a tab genuinely possible rather than a demonstration. What fits is decided almost entirely by arithmetic you can do before writing any code, and by two numbers your own browser will tell you.

The size arithmetic

A model’s weights must fit in GPU memory, and the size of the weights is parameter count multiplied by bytes per parameter. That is the whole first-order calculation, and quantisation is the only lever on the second factor.

weights bytes ≈ parameters × bytes per parameter

  fp16 / bf16   2 bytes
  int8          1 byte
  4-bit        ~0.5 bytes  (plus a small per-block scale; call it ~0.6 in practice)

So:

  1.5B params × 0.6  ≈  0.9 GB
  3B   params × 0.6  ≈  1.8 GB
  7B   params × 0.6  ≈  4.2 GB
  13B  params × 0.6  ≈  7.8 GB

And that is only the weights. Add the KV cache, which grows with
context length:

  kv bytes ≈ 2 × layers × kv_heads × head_dim × bytes_per_element × tokens

  A small model with 28 layers, 4 kv heads, head_dim 128, fp16, at 4k tokens:
    2 × 28 × 4 × 128 × 2 × 4096  ≈  235 MB

Plus the runtime's own buffers, the compiled shaders, and whatever
else the tab is holding.

The second constraint is stricter than total memory and catches people out: a single GPU buffer has a maximum size, reported by the adapter, and it is typically far below the total memory available. A runtime therefore has to split weights across many buffers, and a naively-packed model that wants one giant buffer fails on a device with plenty of free memory. That is why the probe below reports maxBufferSize and maxStorageBufferBindingSize alongside anything else.

The honest summary of what this means: models in the low billions of parameters at 4-bit are the realistic range for a browser tab today, and the ceiling is set by the specific device rather than by the specification. Nobody has run a systematic survey of which models load on which hardware, so this page does not print one — it gives you the arithmetic and a probe, which is a better answer than a table that would be wrong for your machine anyway.

The first load is the real cost

Inference speed is what gets discussed; the download is what users actually experience. It is a straight division and it is worth doing before committing to the approach.

first-load seconds ≈ weights bytes / effective bandwidth

A 0.9 GB model (≈1.5B params at 4-bit):

  100 Mbit/s  →  0.9 × 8 × 1024 / 100   ≈  74 s
   50 Mbit/s  →                          ≈ 147 s
   20 Mbit/s  →                          ≈ 369 s     (over six minutes)

A 4.2 GB model (≈7B params at 4-bit):

  100 Mbit/s  →                          ≈ 344 s     (nearly six minutes)
   20 Mbit/s  →                          ≈ 1720 s    (nearly half an hour)

Those are optimistic: they assume the full advertised bandwidth with no contention. The consequence is a hard product constraint rather than an engineering one. A browser-resident model is viable for an application the user opens repeatedly and where the weights can be cached across sessions in the Cache API or IndexedDB. It is not viable for a landing page, and it is not viable at all on a metered mobile connection, where you would also be spending the user’s data allowance without asking.

A capability probe for your own machine

This is the useful artefact. It is plain WebGPU with no dependencies, it runs in any page, and it prints the numbers that decide what your device can hold. Paste it into a console on the machine you care about.

// probe.ts — no dependencies; run it in the browser you are targeting.
export type Probe =
  | { supported: false; reason: string }
  | {
      supported: true;
      adapter: { vendor: string; architecture: string; description: string };
      maxBufferBytes: number;
      maxStorageBindingBytes: number;
      maxComputeWorkgroupStorage: number;
      f16: boolean;
      estimatedFit: { params4bitBillions: number };
    };

export async function probeWebGpu(): Promise<Probe> {
  if (!("gpu" in navigator)) {
    return { supported: false, reason: "navigator.gpu is undefined" };
  }

  const adapter = await navigator.gpu.requestAdapter({
    powerPreference: "high-performance",
  });
  if (!adapter) {
    return { supported: false, reason: "no adapter (often a blocklisted GPU or driver)" };
  }

  const limits = adapter.limits;
  const info = (adapter as any).info ?? {};

  // The single-buffer ceiling is usually the binding constraint, not total
  // VRAM — and the browser does not tell you total VRAM at all.
  const maxBufferBytes = Number(limits.maxBufferSize ?? 0);
  const maxStorageBindingBytes = Number(limits.maxStorageBufferBindingSize ?? 0);

  // Very rough: at ~0.6 bytes per parameter for 4-bit, how many billion
  // parameters would fit in eight buffers of the maximum size?
  const usableBytes = maxBufferBytes * 8;
  const params4bitBillions = usableBytes / 0.6 / 1e9;

  return {
    supported: true,
    adapter: {
      vendor: info.vendor ?? "unknown",
      architecture: info.architecture ?? "unknown",
      description: info.description ?? "unknown",
    },
    maxBufferBytes,
    maxStorageBindingBytes,
    maxComputeWorkgroupStorage: Number(limits.maxComputeWorkgroupStorageSize ?? 0),
    f16: adapter.features.has("shader-f16"),
    estimatedFit: { params4bitBillions: Number(params4bitBillions.toFixed(1)) },
  };
}

// Usage:
// probeWebGpu().then((p) => console.table(p));

Read three things off the output. If supported is false, this user gets the server path and no amount of code changes that — WebGPU is gated on the browser, the platform and a driver allowlist, so a perfectly capable GPU can still report no adapter. If f16 is false, half-precision compute is unavailable and throughput will be materially worse. And maxBufferBytes is the number to compare your candidate model’s largest tensor against.

The estimatedFit figure is deliberately crude and labelled as such: it multiplies the buffer ceiling by an assumed number of buffers, which is a modelling choice rather than a measurement. Treat it as an order of magnitude, and treat an actual load attempt as the real test.

// The honest way to answer "does it work here": try, and time it.
const started = performance.now();
try {
  await loadModel(candidate, {
    onProgress: (fraction) => {
      const elapsed = (performance.now() - started) / 1000;
      console.log(
        "loaded", Math.round(fraction * 100) + "%",
        "in", elapsed.toFixed(1) + "s",
        "projected total", (elapsed / Math.max(fraction, 0.01)).toFixed(0) + "s",
      );
    },
  });
  console.log("first load complete in", ((performance.now() - started) / 1000).toFixed(1), "s");
} catch (err) {
  console.error("did not load on this device:", err);
}

Loading a model, and what the library does

Nobody writes the inference kernels by hand. Several projects — the best known are the WebLLM and Transformers.js families — package quantised weights plus WebGPU or WebAssembly kernels behind a load-and-generate API. What they all do is the same four things, and knowing the shape is more durable than knowing any one API.

1. Fetch the weights in shards, so progress is reportable and a failure is resumable. Shards also sidestep the single-buffer ceiling.
2. Cache them in the Cache API or IndexedDB, keyed by a model and revision id, so the second visit is fast. This is the step that makes the whole approach viable.
3. Compile shaders and allocate buffers on the device. Slow the first time, cached by the browser afterwards.
4. Run the generation loop, streaming tokens back through a callback — and in a Web Worker, because token generation on the main thread makes the page unresponsive.

When this is worth doing

The genuine reasons, and they are narrower than the enthusiasm suggests:

• The data must not leave the device. Health notes, legal drafts, an internal codebase. This is the strongest reason and frequently the only one that survives a cost comparison — related: local model privacy.
• Offline operation is a requirement. Field tools, aircraft, anywhere connectivity is not assumed.
• Per-request cost must be zero at scale. A free tier with heavy usage where the marginal cost of an API call is the thing killing you. The user pays in battery and in a large download.
• Latency to first token matters more than quality. A loaded local model has no network round trip at all, which for autocomplete-shaped features can beat a far better remote model on perceived responsiveness.

Against those: a small local model is meaningfully weaker than a current hosted one, generation is slower on most consumer hardware, battery drain is real and noticeable on laptops, and you have taken on a device-compatibility matrix. The hybrid is usually the right answer — local for the cheap and private steps, remote for the hard ones — which is the subject of combining local and API models.