Caching Model Responses in Python

An LLM cache is trivial to write and easy to get subtly wrong. The failure is never a crash — it is that you edit a prompt, deploy, and keep serving answers generated by the old one, for as long as the TTL allows.

What goes into the key

The rule is exact: everything that can change the answer goes into the key. Anything you leave out becomes a way to serve a stale result after a change nobody thought was a cache concern.

The neatest way to satisfy all of this at once is to hash the request body you are about to send, because by construction it contains every parameter the provider will act on. Then add the template version and the tenant id, which are not in the body.

The key function

# cache_key.py
import hashlib
import json
from typing import Any

TEMPLATE_VERSION = "2026-08-04.a"   # bump this when you edit a prompt template


def cache_key(payload: dict[str, Any], *, tenant: str | None = None) -> str:
    """A stable hash of everything that changes the answer."""
    material = {
        "payload": payload,
        "template": TEMPLATE_VERSION,
        "tenant": tenant,
    }
    encoded = json.dumps(
        material,
        sort_keys=True,          # dict order must not change the key
        separators=(",", ":"),   # no incidental whitespace
        ensure_ascii=False,
        default=str,             # datetimes and Decimals do not crash the hash
    ).encode("utf-8")
    return hashlib.sha256(encoded).hexdigest()

sort_keys=True is what stops two logically identical requests hashing differently because a dict was built in a different order — a bug that presents as a hit rate of zero with no error anywhere. sha256 rather than Python’s built-in hash() because hash() of a string is salted per process and changes on every restart, which makes a persistent cache useless in a way that takes an hour to notice.

The store

SQLite is the right first answer: one file, no server, survives restarts, and queryable when you want to know what the cache actually contains. Move to Redis when several processes need to share it or when you want the TTL enforced for you.

# cache.py
import json
import sqlite3
import time
from typing import Any

SCHEMA = """
CREATE TABLE IF NOT EXISTS llm_cache (
  key         TEXT PRIMARY KEY,
  value       TEXT NOT NULL,
  model       TEXT NOT NULL,
  created_at  REAL NOT NULL,
  expires_at  REAL,
  hits        INTEGER NOT NULL DEFAULT 0
);
CREATE INDEX IF NOT EXISTS llm_cache_expiry ON llm_cache (expires_at);
"""


class ResponseCache:
    def __init__(self, path: str = "llm_cache.db"):
        self.conn = sqlite3.connect(path, isolation_level=None)
        self.conn.row_factory = sqlite3.Row
        self.conn.execute("PRAGMA journal_mode=WAL")
        self.conn.executescript(SCHEMA)

    def get(self, key: str) -> Any | None:
        row = self.conn.execute(
            "SELECT value, expires_at FROM llm_cache WHERE key = ?", (key,)
        ).fetchone()
        if row is None:
            return None
        if row["expires_at"] is not None and row["expires_at"] < time.time():
            self.conn.execute("DELETE FROM llm_cache WHERE key = ?", (key,))
            return None
        self.conn.execute(
            "UPDATE llm_cache SET hits = hits + 1 WHERE key = ?", (key,)
        )
        return json.loads(row["value"])

    def set(self, key: str, value: Any, model: str, ttl: float | None) -> None:
        now = time.time()
        self.conn.execute(
            "INSERT OR REPLACE INTO llm_cache"
            " (key, value, model, created_at, expires_at, hits)"
            " VALUES (?, ?, ?, ?, ?, 0)",
            (key, json.dumps(value), model, now,
             now + ttl if ttl is not None else None),
        )

    def stats(self) -> dict:
        row = self.conn.execute(
            "SELECT count(*) AS entries, sum(hits) AS hits,"
            " sum(CASE WHEN hits = 0 THEN 1 ELSE 0 END) AS never_used"
            " FROM llm_cache"
        ).fetchone()
        return dict(row)

Wiring it in is one wrapper:

def cached_call(cache: ResponseCache, client, payload: dict,
                *, ttl: float | None = 86400.0, tenant: str | None = None) -> dict:
    key = cache_key(payload, tenant=tenant)
    hit = cache.get(key)
    if hit is not None:
        return hit
    response = client.post("/chat/completions", json=payload)
    response.raise_for_status()
    body = response.json()
    cache.set(key, body, payload["model"], ttl)
    return body

Cache the whole response body, not just the text. The usage object is what makes the saving measurable — a hit is tokens you did not buy, and without storing the counts you cannot say how many.

The stats() query is worth running weekly. A large never_used count means you are caching requests that never repeat, which is disk and complexity spent for nothing; the answer then is a smaller cache scoped to the calls that do repeat.

Choosing a TTL by asking one question

The question is: how long would a wrong answer be acceptable? Not how long the answer stays true — how long you could serve a stale one before somebody minds. That reframing gives an answer immediately where “how volatile is this data” produces an argument.

Two mechanics that matter more than the number. First, a TTL is an upper bound, not a schedule: a template edit should invalidate immediately, which is what TEMPLATE_VERSION is for. Second, if many entries are written together they expire together, and the resulting stampede hits your provider all at once — spread the writes with a jittered TTL, ttl * random.uniform(0.9, 1.1), exactly as with any other cache.

Deciding whether the cache is worth it

A cache is worth building when the repeat rate is high enough to pay for the complexity, and that is measurable before you write any of the code above. You need one number: how many of your requests are exact repeats of an earlier one.

1. Hash without caching. Add cache_key(payload) to your call log — the prompt_sha field in logging every model call is exactly this — and change nothing else. A day of real traffic is usually enough.
2. Count the duplicates. SELECT count(*) - count(DISTINCT prompt_sha) FROM calls over that day is the number of calls a perfect cache would have eliminated.
3. Price them. Sum the recorded cost of every call whose hash had been seen before. That is the saving, in the currency you are billed in, with no modelling assumptions.
4. Check the shape, not just the total. A saving concentrated in one hash — a health check, a fixed prompt in a cron job — is better addressed by not making that call. A saving spread over thousands of hashes is the case a cache is for.

-- what a perfect cache would have saved, over one day of logs
SELECT
  count(*)                                   AS calls,
  count(DISTINCT prompt_sha)                 AS distinct_prompts,
  100.0 * (1 - count(DISTINCT prompt_sha)::float / count(*)) AS repeat_pct,
  sum(cost_micros) / 1e6                     AS spent,
  (sum(cost_micros) - sum(first_cost)) / 1e6 AS saveable
FROM (
  SELECT prompt_sha, cost_micros,
         first_value(cost_micros) OVER (PARTITION BY prompt_sha
                                        ORDER BY started_at) AS first_cost
  FROM calls WHERE status = 'ok'
);

Two thresholds are worth stating plainly. Below about a five per cent repeat rate, the cache is a store of prompts, a retention obligation and an invalidation bug waiting to happen, in exchange for very little — normalise the prompts and measure again before building it. Above twenty, it is one of the cheapest optimisations available, and the latency win on a hit is usually more valuable than the money.

Once it is running, keep watching hits and never_used from stats(). A hit rate that falls without an explanation almost always means something entered the key that varies per request — a timestamp in the system prompt, a randomly ordered list of retrieved documents, a session id. That is a bug in what you are sending, and the cache is the only place it shows up.

What must never be cached

• Anything sampled at a high temperature for variety. If the user pressed “regenerate”, returning the cached answer is a bug that looks like the button being broken. Include a nonce in the key or skip the cache on that path.
• Streamed responses, as streams. You can cache the assembled text and replay it, but a replayed “stream” arriving in a millisecond is a different user experience. Cache the text; decide deliberately whether to fake the typing.
• Errors. A 429 or a 503 is not an answer. Caching them turns a transient failure into a persistent one for the length of the TTL.
• Anything keyed without the tenant. The worst possible cache bug is one customer receiving another customer’s answer. If the key does not include the tenant, the cache must be per-tenant at the storage level instead.

When exact-match caching is not enough

The cache above hits only on a byte-identical request. For free-text questions that is a hit rate near zero, because “how do I reset my password” and “password reset?” hash differently.

Two routes out, in increasing order of risk. Normalise before hashing — lowercase, collapse whitespace, strip trailing punctuation — which is cheap, safe and buys a surprising amount. Or embed the question and treat a nearest neighbour above a similarity threshold as a hit, which is semantic caching and which trades correctness for hit rate: at a threshold loose enough to be useful, some questions get answers to a different question. Set the threshold with a labelled set of real query pairs, not by intuition.

Finally, note that a response cache is not the same thing as prompt caching, which is a provider-side discount on re-processing a shared prefix. The two compose: yours removes the call entirely, theirs makes the calls you do make cheaper.