Timeout Chooser
Paste your own latency samples and get percentiles, a timeout derived from p99, the share of requests it would cut, and the worst case once retries are included.
p99 of 3,050 ms × 1.50. 0 of 30 pasted samples (0.00%) would have been cut at this value.
- Samples parsed
- 30
- Minimum
- 760 ms
- p50
- 990 ms
- p90
- 1,340 ms
- p95
- 2,100 ms
- p99
- 3,050 ms
- Maximum
- 3,050 ms
- Recommended timeout
- 4,575 ms
- Requests it would cut
- 0.00%
- Worst case over 3 attempt(s)
- 15.72 s
A timeout is a decision about the tail
timeout = p99 × safety multiplier
Most timeouts are round numbers someone typed: thirty seconds, sixty, whatever the client library defaulted to. That is a guess about your latency distribution made by someone who never saw it. Deriving the number from the distribution takes a paste and a multiplication, and it converts an argument into arithmetic — you can say precisely what fraction of real requests the value would have killed.
Set it too low and you cut requests that were going to succeed, and since the model has already done the expensive part, you pay for the work and throw the answer away — then usually retry, doubling the cost of a request that was merely slow. Set it too high and a hung connection holds a concurrency slot for the entire duration, which at LLM concurrency levels is how one stuck upstream turns into a queue that swallows the whole pool.
Two adjustments the number above cannot make for you. Streaming responses want two separate timeouts — a short one for time to first token, which tells you whether the request was accepted, and a longer idle timeout between chunks, which catches a stream that died mid-answer. And whatever you choose interacts with retries: the worst-case row is what a user actually experiences before your code gives up, and it is the number to compare against your own latency promise, not the single-attempt timeout.