WebLLM: Start to First Response in the Browser

WebLLM compiles a quantized model to WebGPU shaders and runs the whole generation loop inside the tab. No server, no token leaving the machine, and an OpenAI-shaped API on top. The part that decides whether it is a good idea is not the API — it is the several hundred megabytes to several gigabytes you are asking a visitor to download before anything happens.

What has to be true first

WebLLM has one hard dependency and it is not negotiable: WebGPU. There is no CPU fallback inside the library. If navigator.gpu is missing, the engine throws WebGPUNotFoundError with the message Cannot find WebGPU in the environment; if the adapter request fails, you get WebGPUNotAvailableError, whose text begins WebGPU is not supported in your current environment, but it is necessary to run the WebLLM engine. Those two strings are worth recognising because they are the entire failure surface for “it works on my machine”. What to do about them is a page of its own: what happens when a browser does not support WebGPU.

WebGPU also requires a secure context, so this works on https:// and on http://localhost and nowhere else. A second precondition is subtler. Half of WebLLM’s prebuilt catalogue is quantized to q4f16_1, and those entries carry required_features: ["shader-f16"] in the library’s own prebuiltAppConfig. On an adapter without that WebGPU feature you get ShaderF16SupportError, and the fix is to pick the q4f32_1 build of the same model rather than to chase browser flags.

The download is the decision

WebLLM publishes a vram_required_MB figure for every entry in its prebuilt config, and reading it before you choose a model is the single highest-value thing on this page. In the config on the project’s main branch, Llama-3.2-1B-Instruct-q4f16_1-MLC declares 879.04 MB and is flagged low_resource_required: true; Llama-3.2-3B-Instruct-q4f16_1-MLC declares 2263.69 MB; and Llama-3.1-8B-Instruct-q4f16_1-MLC declares 5001.0 MB. Those are the library’s numbers for the resident footprint, not a measurement of your GPU.

Weights are cached in the browser’s Cache API after the first load, so the cost falls on the first visit and on anyone whose storage gets evicted. This is why a 1B model is almost always the right starting point for a demo and an 8B model is almost never the right thing to put on a public marketing page: you are spending a visitor’s bandwidth before they have seen anything work.

Getting to the first token

1. Install the package: npm install @mlc-ai/web-llm. It is an ES-module browser package; there is nothing to install on a server.
2. Serve the page over HTTPS or from localhost, so the secure context requirement is satisfied.
3. Create an engine, giving it a progress callback so the first-load download is visible rather than a frozen page.
4. Call engine.chat.completions.create with an OpenAI-shaped request.

import * as webllm from "@mlc-ai/web-llm";

const selectedModel = "Llama-3.2-1B-Instruct-q4f32_1-MLC";

const engine = await webllm.CreateMLCEngine(
  selectedModel,
  {
    initProgressCallback: (report: webllm.InitProgressReport) => {
      document.getElementById("init-label")!.innerText = report.text;
    },
    logLevel: "INFO",
  },
  // KV cache sizing: a smaller window is a smaller allocation.
  { context_window_size: 2048 },
);

const reply = await engine.chat.completions.create({
  messages: [{ role: "user", content: "List three US states." }],
  max_tokens: 256,
});

console.log(reply.choices[0].message.content);
console.log(reply.usage);

The third argument is a ChatOptions override and it matters more here than it would against a hosted API. The KV cache is allocated as GPU buffers up front from context_window_size, so halving the window halves that allocation. On a machine that is close to the edge, this is the difference between a model that loads and a DeviceLostError, whose message names the cause directly: The WebGPU device was lost while loading the model. This issue often occurs due to running out of memory. The related constraint — the maximum size of any single buffer WebGPU will let you bind — is covered in WebGPU browser model size limits.

Streaming and the usage object

Non-streaming is fine for a first run and wrong for anything a person watches, for the same reason it is wrong against a hosted API: the reader waits for the whole answer instead of for the first token. Set stream: true and iterate the async generator. Ask for usage explicitly with stream_options, because only the final chunk carries it.

const chunks = await engine.chat.completions.create({
  stream: true,
  stream_options: { include_usage: true },
  messages: [{ role: "user", content: "Provide me three US states." }],
  temperature: 1.0,
  max_tokens: 256,
});

let message = "";
for await (const chunk of chunks) {
  message += chunk.choices[0]?.delta?.content || "";
  render(message);
  if (chunk.usage) console.log(chunk.usage); // last chunk only
}

The usage object on the final chunk is where WebLLM reports prefill and decode throughput for that request. It is the only honest source of tokens per second for this combination, because nobody publishes a table of browser inference speeds that would apply to your visitor’s GPU, driver and thermal state. If you want the number, read it from usage on the machine in question.

Move it off the main thread

Generation on the main thread will make the page janky — not because the shaders run there, but because the scheduling, the token sampling and the tokenizer work do. WebLLM ships CreateWebWorkerMLCEngine for exactly this, and it takes the same model id and the same options object, so it is a two-line change once the rest works.

const engine = await webllm.CreateWebWorkerMLCEngine(
  new Worker(new URL("./worker.ts", import.meta.url), { type: "module" }),
  "Llama-3.2-1B-Instruct-q4f32_1-MLC",
  { initProgressCallback },
);

The reason this is not optional for anything real is that the load itself is the worst offender, not the generation. Compiling shaders and uploading several hundred megabytes of weights to the GPU happens while your page is trying to render a progress indicator, and on the main thread the progress indicator is the thing that stops updating. Moving the engine to a worker means the callback you wired up in the first section actually animates.

The worker file itself is a handler that forwards messages to an MLCEngine; the shape is in the project’s basic usage documentation. There is also a service-worker variant, which keeps the model loaded across page navigations — useful, and a good way to hold a multi-gigabyte allocation open longer than a user expects, so use it deliberately.