Cache Hit Rate Estimator

Estimates the hit rate a cache of a given size earns under a stated Zipf popularity model, and what that is worth against your own per-request costs.

Requests per month

Distinct cacheable keysHow many different prompts, prefixes or queries exist at all.

Entries the cache holdsCapacity in keys, not bytes.

Popularity skewThe Zipf exponent. 0 = every key equally likely; 0.7–1.0 is the range usually fitted to web and query traffic; above 1.2 means a handful of keys are almost all of it.

Share of traffic that is cacheable at allRequests with a stable key. Anything with a timestamp, a user id or a random sample in the prompt is not.

Cost of a missExample figure — your own cost per uncached request.

$ / req

Cost of a hitExample figure — a cached read is cheaper, not free.

$ / req

Hit rate this cache size earns

48.8%

Holding 10.0% of the key space covers 69.8% of cacheable traffic at skew 0.90. Drop the skew to 0.5 and watch the answer collapse — that is how much of it rests on the assumption.

Cacheable requests per month: 1,400,000
Keys resident: 5,000
Coverage of cacheable traffic: 69.8%
Hits per month: 976,816
Misses per month: 1,023,184
Monthly cost with no cache: $8,000
Monthly cost with this cache: $5,070
Saving: $2,930
Effective cost per request: $0.0025

What this assumes: requests are drawn independently from a Zipf distribution over the key space — the independent reference model — and the cache holds the most popular keys, which is where an LRU settles once the working set is stable. There is no TTL: an entry stays until something more popular displaces it. A cache with a short lifetime, which is what a provider prompt cache measured in minutes is, gets less than this; model that by lowering the cacheable share rather than by trusting the coverage figure. No warm-up period, and no correlation between requests beyond popularity, so a burst of identical requests does better than this predicts.

Why a hit rate is a property of the traffic, not the cache

Nobody can tell you your cache hit rate, and any tool that prints one without telling you what it assumed is guessing on your behalf. What can be derived is the relationship: given a key space of a certain size, a popularity distribution of a certain shape, and a cache that holds a certain number of entries, the share of requests that find something is the fraction of total probability mass sitting in the top entries. Under a Zipf distribution that is a ratio of two harmonic numbers, which is what this computes.

The useful consequence is how non-linear it is. When the skew is near 1, popularity falls off like 1/rank and coverage grows like the logarithm of capacity — so the first 1% of the key space buys a large share of the traffic and the next 10% buys much less. That is the honest argument for a small cache, and it is also the argument against buying a bigger one when the current one is already past the knee. When the skew is near 0, traffic is uniform, coverage is just capacity over key space, and caching is close to pointless.

Two things are outside this model and both make the real number worse. Time: entries expire, and a key that repeats once a week never hits a cache with a five-minute lifetime however popular it is. Invalidation: the moment the underlying data changes, every entry derived from it should be gone, and how aggressively you do that is a correctness decision that costs hit rate. Measure the real thing once you are live; use this to decide whether it is worth building.

Cache Hit Rate Estimator

Why a hit rate is a property of the traffic, not the cache

The reasoning behind this

Related tools