Testing Thread Safety of a Shared LLM Client Instance

One client object, constructed at import time, shared by every request handler. Somebody adds a per-request header for tenant attribution by setting it on the client. Under load, one tenant’s requests start carrying another tenant’s identifier, and nothing in the logs looks wrong.

The shape of the leak

The bug is never described as a concurrency bug when it is reported. It arrives as “user A saw user B’s data”, or as a billing report attributing spend to the wrong account, or as traces where a span belongs to the wrong request. The mechanism is always the same: mutable state on a long-lived object, written by one request and read by another between the write and the send.

The candidates are few enough to enumerate, which is what makes this testable rather than open-ended.

• Default headers set per request. Authorization for a caller-supplied key, a tenant id, a trace id, an idempotency key. Setting any of these on a shared client rather than passing them per call is the classic form.
• A mutable default argument or module-level accumulator. A conversation history list, a token counter, a retry count. In Python a mutable default argument is shared by every call to the function, forever, which produces this bug with no threads involved at all.
• A base URL or model swapped for one request. Common when someone adds a “use the cheap model for this one path” feature by mutating the client.
• A cached prompt or system message on the client. Efficient, and correct only if it is genuinely global.

Most vendor SDK clients are safe to share; it is the wrapper written around them that is not. That is worth stating because the instinct on reading a bug like this is to stop sharing the client, which costs you connection pooling for no reason.

A harness that forces the interleaving

Firing a thousand threads and hoping is a slow, flaky test. Force the interleaving instead: every thread sets its own value, then waits at a barrier until all of them have set theirs, and only then reads. If the state is shared, every thread reads the last write and all but one assertion fails, every single run.

# test_client_thread_safety.py
import threading
from concurrent.futures import ThreadPoolExecutor

def test_tenant_header_does_not_leak_between_threads(client, capture_server):
    n = 16
    barrier = threading.Barrier(n)

    def one(i):
        tenant = "tenant-%d" % i
        with client.scoped(tenant_id=tenant):     # the API under test
            barrier.wait(timeout=5)               # every thread has now set its value
            resp = client.complete(prompt="ping")
        return i, capture_server.header_for(resp.request_id, "x-tenant-id")

    with ThreadPoolExecutor(max_workers=n) as pool:
        results = list(pool.map(one, range(n)))

    for i, seen in results:
        assert seen == "tenant-%d" % i, "thread %d sent %s" % (i, seen)

The barrier is what turns a probabilistic test into a deterministic one. Without it, sixteen threads may well run to completion one after another and the test passes on a broken client. With it, the window during which the state is wrong is guaranteed to be open when the read happens.

The capture server is a local HTTP server that records the headers of each request against an identifier echoed back in the response. Record server-side rather than asserting on the client’s own view of what it was about to send: what matters is the bytes that left the process, and a client that reports the right header while sending the wrong one is exactly the failure mode you are hunting.

What to assert, and what a pass means

Assert per-thread identity, not aggregate counts. A test asserting that sixteen distinct tenant values were seen passes when the values are all present but attached to the wrong requests, which is the leak. The assertion has to pair each request with the value the thread that made it intended.

Three more assertions are worth having in the same file: that no request left with a missing or empty tenant header, that no request carried a value belonging to a thread that had already finished, and — if you accumulate usage — that the per-tenant token totals sum to the global total. That last one catches the counter version of the bug, where nothing is misattributed but increments are lost to a non-atomic read-modify-write.

Async is not exempt

Single-threaded async runtimes make this bug easier to believe you do not have, and they have it in a subtler form. There is no preemption mid-statement, so a read-modify-write of a counter is safe — but every await is a yield point, and mutable state set before an await and read after it can be overwritten by another task in between. Setting a header on a shared client and then awaiting the request is precisely that pattern.

The equivalent harness replaces threads with tasks and the barrier with an event, and it is materially easier to write because you control the scheduling: run the tasks concurrently, and make the fake transport yield once before it reads the header. In Python the fix is usually a ContextVar, which is per-task rather than per-thread and propagates correctly across awaits; in Node it is the async local storage API. Both are worth a test asserting the value survives an await boundary and does not survive into a sibling task.

The structural test that actually holds

Since the race test can pass on broken code, add a test that cannot. The design rule is that per-request state never lives on the shared object, and that rule is checkable directly.

1. Freeze the client after construction, or make the wrapper expose no setters at all. Then assert in a test that attempting to set a request-scoped field on the shared instance raises. A compile-time equivalent — a readonly type, a frozen dataclass — is better still, because it fails before the test runs.
2. Assert that the per-call API carries everything scoped. Inspect the signature of your complete wrapper and assert the scoped fields are parameters, so a future refactor that moves one onto the client fails a test with an explanatory name.
3. Assert the shared client is genuinely shared. If someone “fixes” this by constructing a client per request, the leak goes away and connection reuse goes with it. A test asserting the factory returns the same instance twice records that this was a decision.
4. Run the race test at a high worker count in CI but keep it off the fast unit path, and let it be the thing that fails loudly when someone reintroduces mutable state. Record which thread saw which value in the failure output; a bare assertion failure on a concurrency test is nearly impossible to act on, and what you record is what makes it diagnosable.