Testing Cancellation of an In-Flight Streaming Request

Pressing stop always looks like it worked, because the UI stops updating either way. Whether it did anything depends on a chain of three connections, and the only way to know is to have the far end tell you it noticed.

What cancelling has to mean

A user clicking Stop in a browser is asking for four things, and code commonly delivers only the first:

1. The UI stops appending tokens and returns to an idle state.
2. The browser’s connection to your endpoint is closed, rather than left reading into a discarded buffer.
3. Your endpoint notices that close and closes its connection to the provider.
4. Your accounting records what was actually consumed, because the usage frame that normally reports it will not arrive.

Steps one and two are usually free: AbortController plus a signal on fetch gives you both, and the reader rejects with an AbortError. Step three is the one that silently does not happen, and it is the expensive one — a server that keeps reading an abandoned stream is generating and paying for tokens that no user will ever see, for as long as the model keeps talking.

Forwarding the signal

In any runtime built on the Fetch standard, the incoming Request carries a signal that aborts when the client disconnects. Forwarding it to the upstream fetch is one line, and its absence is the defect this whole page is about.

export async function POST(request: Request) {
  const upstream = await fetch(PROVIDER_URL, {
    method: "POST",
    headers: {
      "content-type": "application/json",
      authorization: "Bearer " + process.env.PROVIDER_KEY,
    },
    body: await request.text(),
    signal: request.signal,        // <- the line this test protects
  });

  return new Response(upstream.body, {
    headers: {
      "content-type": "text/event-stream",
      "cache-control": "no-cache",
      connection: "keep-alive",
    },
  });
}

On a Node http server the equivalent is listening for close on the request and calling abort() on your own controller, or using pipeline, which destroys the source when the destination goes away. If you are transforming frames rather than passing the body straight through, the transform must propagate the cancel too — a manual for await loop over the upstream body with a try / finally that calls reader.cancel() is the minimum.

The test that proves it

The assertion has to be made at the upstream, because that is where the evidence is. A fixture server that records whether its request was closed early gives you a boolean to assert on.

import { createServer } from "node:http";
import { once } from "node:events";
import { expect, test, vi } from "vitest";

test("aborting the client request closes the provider connection", async () => {
  let upstreamClosedEarly = false;
  let framesWritten = 0;
  let timer: NodeJS.Timeout;

  const provider = createServer((req, res) => {
    res.writeHead(200, { "content-type": "text/event-stream" });
    timer = setInterval(() => {
      framesWritten++;
      res.write('data: {"choices":[{"index":0,"delta":{"content":"tok "}}]}\n\n');
    }, 10);
    req.on("close", () => {
      clearInterval(timer);
      if (!res.writableEnded) upstreamClosedEarly = true;
    });
  });
  provider.listen(0);
  await once(provider, "listening");

  const controller = new AbortController();
  const res = await fetch(myEndpoint, {
    method: "POST",
    body: JSON.stringify({ messages: [{ role: "user", content: "count" }] }),
    signal: controller.signal,
  });

  const reader = res.body!.getReader();
  await reader.read();                       // wait for the first real frame
  const framesAtAbort = framesWritten;
  controller.abort();

  await vi.waitFor(() => expect(upstreamClosedEarly).toBe(true), { timeout: 2000 });

  // and it stopped producing, rather than merely being ignored
  await new Promise((r) => setTimeout(r, 100));
  expect(framesWritten).toBeLessThan(framesAtAbort + 3);

  provider.close();
});

Two assertions, doing different jobs. upstreamClosedEarly proves the disconnect propagated. The frame-count check proves it propagated promptly — a handler that only notices on its next read, or that has buffered a hundred frames ahead, will fail the second while passing the first. Keep the tolerance loose; a frame or two in flight at the moment of abort is normal and is not a bug.

If your endpoint transforms frames rather than piping the body straight through, add a third assertion: that your handler’s own cleanup ran. The open-handle counter from the clean-close test works unchanged here, and this is the path where it usually fails — a finally that clears a keepalive interval on success is very often missing from the abort branch, which leaves one timer per cancelled request writing into a socket that no longer exists.

Accounting for a cancelled stream

Cancellation breaks your cost accounting in a specific way that is worth a test of its own. With stream_options: {"include_usage": true}, the token totals arrive on one extra chunk at the very end of the stream — and OpenAI’s own documentation notes that if the stream is interrupted or cancelled you may not receive that final usage chunk. The same logic applies to Anthropic’s message_delta, which is where cumulative output tokens are reported.

So a cancelled request produces cost with no usage record. If your spend tracking only writes a row when the usage frame arrives, every cancelled request is invisible in your numbers, and the gap grows with however aggressively your users hit stop. Assert the behaviour you want instead:

• A cancelled stream still writes a usage row, flagged as estimated.
• The estimate is derived from what you received — the prompt you sent, plus a token count over the deltas that arrived. Label it as an estimate in the data, not just in a comment, so a later report can separate measured from estimated spend.
• Two cancelled requests do not produce one row. Idempotency on the accounting write matters here because the abort path and the error path can both fire.

Not treating an abort as a failure

The last assertion is about noise. An intentional cancel must not look like an outage: it should not increment your error rate, should not trigger a retry, and should not page anybody. The rejection from an aborted fetch is a DOMException with name === "AbortError", and that check is what your error handler needs to branch on — not the message, which differs across runtimes.

Assert it directly: after a cancel, your error counter is unchanged, your retry function was not called, and the log line written is at info level with a distinguishable reason. This is the difference between a stream that dropped and a stream somebody stopped, and code that conflates them will retry requests the user explicitly abandoned.