Testing That Concurrent Requests to One Cache Key Don't Stampede

A popular document is uncached for one second — a deploy, an eviction, a TTL expiry — and in that second two hundred requests arrive for it. All two hundred miss, all two hundred call the model, and you pay for two hundred identical generations while the provider rate-limits you for the trouble.

The symptom

It looks like a spike rather than an error, which is why it takes a while to identify. Characteristic signs: cost per request that is fine on average and terrible at p99; a burst of 429s clustered in time rather than spread; a cache hit rate that is high overall and drops to zero for a fraction of a second periodically; identical requests with near-identical timestamps in the provider’s logs. The classic trigger is a deploy that starts with an empty cache while traffic is already at full rate.

Why a correct cache still stampedes

The standard read-through cache is a two-step: look up, and on a miss, compute and store. There is no defect in that logic. The problem is that the interval between the lookup and the store is not zero — for a model call it is seconds — and every request arriving in that interval sees the same empty cache and correctly concludes it must compute.

Concurrency multiplies the damage by the request rate, so this gets worse exactly as the system gets more successful. And it is worst for the most valuable keys: a key nobody wants has no stampede, because there is only ever one request for it. The keys that stampede are the hot ones you are caching precisely because they are hot.

Two adjacent problems often get conflated with it. Synchronised expiry, where many keys share a TTL and all expire at once, is a different fault with a different fix (jitter the TTLs). And a cache key that does not distinguish two different prompts is a correctness bug, not a load one. This page is only about many concurrent misses on one legitimately shared key.

Single-flight, and its own bug

The fix is to cache the promise rather than the value: the first caller to miss stores its in-flight work under the key, and subsequent callers find it and await the same operation. Everybody gets the same answer and the expensive call happens once.

// src/single-flight.ts
const inFlight = new Map<string, Promise<string>>();

export async function getOrGenerate(
  key: string,
  cache: Cache,
  generate: () => Promise<string>,
): Promise<string> {
  const hit = await cache.get(key);
  if (hit !== undefined) return hit;

  const existing = inFlight.get(key);
  if (existing) return existing;

  const work = (async () => {
    const value = await generate();
    await cache.set(key, value);
    return value;
  })();

  inFlight.set(key, work);
  try {
    return await work;
  } finally {
    // The line the naive version omits.
    inFlight.delete(key);
  }
}

That finally is the whole reason this page is a fix rather than a tip. Without it, a rejected promise stays in the map forever, and every subsequent caller for that key awaits a promise that is already rejected — so one transient provider error becomes a permanent, un-retryable failure for that key that survives until the process restarts. It is a worse bug than the stampede, it is caused by fixing the stampede, and the obvious test (“does it call the model once?”) passes with it present.

There is a second decision hiding here. Should the joined callers share the failure, or should each retry independently? Sharing is usually right — if the provider is refusing, two hundred simultaneous retries are the stampede again — but it means one caller’s bad luck fails all of them, so the error must be recognisable as a shared one and the retry must happen at a layer that knows to back off.

The test

Concurrency here means overlapping promises, not threads, so the harness is simply firing N calls without awaiting between them. Keep the generator suspended until every caller has arrived, which is what makes the test deterministic rather than a race you usually win.

import { describe, it, expect, vi } from "vitest";
import { getOrGenerate } from "../src/single-flight";

function deferred<T>() {
  let resolve!: (v: T) => void;
  let reject!: (e: unknown) => void;
  const promise = new Promise<T>((res, rej) => { resolve = res; reject = rej; });
  return { promise, resolve, reject };
}

describe("cache stampede", () => {
  it("calls the model once for 200 concurrent misses", async () => {
    const gate = deferred<string>();
    const generate = vi.fn(() => gate.promise);
    const cache = new MemoryCache();

    const calls = Array.from({ length: 200 }, () =>
      getOrGenerate("doc:42", cache, generate));

    // Every caller has now missed and joined. Only then release the work.
    gate.resolve("the summary");
    const results = await Promise.all(calls);

    expect(generate).toHaveBeenCalledTimes(1);
    expect(new Set(results)).toEqual(new Set(["the summary"]));
    expect(await cache.get("doc:42")).toBe("the summary");
  });

  it("does not poison the key when the first attempt fails", async () => {
    const first = deferred<string>();
    const generate = vi
      .fn()
      .mockReturnValueOnce(first.promise)
      .mockResolvedValueOnce("recovered");
    const cache = new MemoryCache();

    const failing = Promise.all([
      getOrGenerate("doc:42", cache, generate),
      getOrGenerate("doc:42", cache, generate),
    ]);
    first.reject(new Error("upstream 503"));
    await expect(failing).rejects.toThrow("upstream 503");

    // A later caller must get a fresh attempt, not the dead promise.
    await expect(getOrGenerate("doc:42", cache, generate)).resolves.toBe("recovered");
    expect(generate).toHaveBeenCalledTimes(2);
  });
});

The second case is the one to write first if you only write one. Add a third if your cache is negative-caching: assert that a failure is not stored as a value, or you have converted a transient error into a cached one with a TTL attached to it.

Two details keep this test honest as the code around it changes. The number two hundred is not load testing and should not be tuned like it; it is there because a single-flight bug is invisible at two callers and obvious at two hundred, and the test costs the same either way since nothing does real work. And resist adding a timing assertion. The property under test is a call count, which is exact; adding “and it completed within 50ms” converts a deterministic test into one that fails on a loaded CI runner and teaches the team to rerun red builds.

Across processes it is a different problem

The map above is per-process. Eight pods behind a load balancer give you eight concurrent generations instead of two hundred, which is a large improvement and not a solution — and the gap widens every time you scale out.

The cross-process version needs a shared lock: one instance claims the key with an atomic set-if-absent and a short expiry, generates, and writes the value; the others poll the cache briefly and either find the value or, on lock timeout, generate anyway rather than waiting forever. Two properties matter and both need testing. The lock must expire, or a crashed holder blocks the key indefinitely. And the waiters must have a bounded wait with a fallback, or a slow generation converts a stampede into a latency incident.

That test needs a real store — the atomicity is the thing under test, and a fake that implements set-if-absent in JavaScript proves nothing about the store you deploy against. Run it against a container in CI and accept that it is a slower, separate suite. It is also worth asserting the interaction with your rate limiter rather than assuming it: a stampede that is coalesced correctly should not produce a burst at the limiter at all, which is a cleaner signal than counting model calls. See rate limiting an AI endpoint for that side, and what prompt caching actually saves for the provider-side cache, which is a separate layer with its own stampede behaviour.