Calling Models From Cloudflare Workers

A Worker is close to an ideal home for a model proxy: it starts in single-digit milliseconds, it runs near the user, and its runtime is web-standard, which is what your streaming code already expects. The two things that catch people are both about resources, and one of them is measured in a unit most people assume is a different unit.

The Worker

// src/index.ts
export interface Env {
  LLM_API_KEY: string;          // wrangler secret put LLM_API_KEY
}

const CORS = {
  "Access-Control-Allow-Origin": "https://your-app.example",
  "Access-Control-Allow-Headers": "Content-Type",
  "Access-Control-Allow-Methods": "POST, OPTIONS",
};

export default {
  async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
    if (request.method === "OPTIONS") {
      return new Response(null, { status: 204, headers: CORS });
    }
    if (request.method !== "POST") {
      return new Response("method not allowed", { status: 405, headers: CORS });
    }

    const { prompt } = (await request.json()) as { prompt?: unknown };
    if (typeof prompt !== "string" || prompt.length > 4000) {
      return Response.json({ error: "bad prompt" }, { status: 400, headers: CORS });
    }

    const upstream = await fetch("https://api.multigrid.ai/v1/chat/completions", {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        Authorization: "Bearer " + env.LLM_API_KEY,
      },
      body: JSON.stringify({
        model: "openai/gpt-4o-mini",
        messages: [{ role: "user", content: prompt }],
        max_tokens: 800,
        stream: true,
      }),
      signal: request.signal,
    });

    if (!upstream.ok || !upstream.body) {
      return Response.json(
        { error: "upstream", status: upstream.status },
        { status: 502, headers: CORS },
      );
    }

    // Pass the body straight through. No buffering, and no CPU time spent.
    return new Response(upstream.body, {
      headers: {
        ...CORS,
        "Content-Type": "text/event-stream; charset=utf-8",
        "Cache-Control": "no-cache, no-transform",
      },
    });
  },
} satisfies ExportedHandler<Env>;

# wrangler.toml
name = "llm-proxy"
main = "src/index.ts"
compatibility_date = "2026-01-01"

# nodejs_compat is only needed if a dependency reaches for a Node built-in.
# Leave it off if you can; it exists to make ports work, not as a default.
# compatibility_flags = ["nodejs_compat"]

compatibility_date is the version pin for the runtime itself. Setting it and leaving it alone is how a Worker keeps behaving the same way after the platform changes; moving it forward is a deliberate upgrade you test.

CPU time is not wall-clock time

This is the most useful thing to understand about the platform, and it is what makes Workers viable for model proxying at all.

Workers are limited and billed on CPU time — milliseconds your JavaScript actually spends executing. Time spent awaiting a network response is not CPU time. The isolate is not running during the await, and doing nothing costs nothing.

A 40-second streaming completion through the Worker above:

  parse the request body                   ~0.4 ms
  build the upstream request               ~0.1 ms
  await fetch(...)                          0    ms CPU   (40 s wall clock)
  construct the Response                   ~0.1 ms
  stream body pass-through                  0    ms CPU   (no JS per chunk)
                                          ---------
  total CPU                                ~0.6 ms

The same Worker, but parsing every SSE event in JavaScript
to count tokens, at 800 events:

  800 × (decode + split + JSON.parse)      ~8–25 ms CPU

The wall-clock time is identical. The CPU time is 15–40× higher.

The design rule falls straight out: a Worker can wait for a model as long as the connection lives; what it cannot do is think for a long time. Passing a stream through costs essentially nothing. Parsing every event to count tokens costs real CPU. Running a tokeniser over a long prompt in JavaScript costs a great deal, and is the usual cause of a Worker that exceeds its CPU limit.

The subrequest budget

Every outbound fetch from a Worker is a subrequest, and there is a cap per invocation. It is generous for a proxy and tight for anything agentic, which is exactly the workload people move to Workers and then get surprised by.

Subrequests in one invocation, counted:

  A simple proxy
    1 × model call                                        = 1

  RAG with reranking
    1 × embed the query
    1 × vector search (if it is an HTTP service)
    1 × rerank
    1 × generate                                          = 4

  An agent loop, 8 turns, 2 tool calls per turn
    8 × model call
   16 × tool call (each an HTTP fetch)                    = 24

  A batch job fanning out over 200 documents
  200 × model call                                        = 200  ← the one that breaks

The first three are comfortable under any current limit. The fourth is the wrong shape for a Worker regardless of what the limit happens to be, because a single invocation is the wrong unit for a batch job. Fan out over a queue, where each message is its own invocation with its own budget, or use Durable Objects to hold the coordination state. That restructuring is worth doing before you hit the ceiling.

Two things that do not count against the budget, and surprise people in the other direction: reading from a KV or Durable Object binding is not an HTTP subrequest, and neither is the response you return. Bindings are why a Worker that looks network-heavy can have a very small subrequest count.

Secrets, bindings and local development

1. npm create cloudflare@latest llm-proxy, choosing the Hello World Worker with TypeScript.
2. npx wrangler secret put LLM_API_KEY and paste the key. It is encrypted at rest and injected into env at runtime. Never put it in wrangler.toml, which is a committed file.
3. For local development put it in .dev.vars, which wrangler reads and which belongs in .gitignore.
4. npx wrangler dev runs the real runtime locally rather than an emulation. That matters: a Node-only API which would fail on deploy fails here too, which is what you want.
5. npx wrangler deploy, then npx wrangler tail to watch live logs including uncaught exceptions.

Doing work after the response

An invocation normally ends when the response is returned, killing anything still pending — a usage log, a cache write, an analytics ping. ctx.waitUntil() extends the lifetime until the promise settles, without holding up the response.

ctx.waitUntil(
  fetch("https://logs.your-app.example/usage", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ model, prompt_chars: prompt.length, at: Date.now() }),
  }).catch(() => {}),                      // never let logging fail the request
);

return new Response(upstream.body, { headers });

Two rules. The promise must be handed to waitUntil before you return, not after — there is no “after”. And it counts as a subrequest, so a logging call inside an agent loop is part of the budget above.

The traps