System Design Interviews for AI Systems: Three Worked Designs

A classic system design round is about scale: shard it, cache it, queue it. An AI system design round is about uncertainty — the component in the middle is probabilistic, and a design that does not say what happens when it is wrong has not answered the question.

What is different about this round

The familiar moves still apply, and you should still make them: estimate the load, name the storage, draw the request path. But four concerns arrive that a normal design round does not have, and candidates who miss all four rarely pass however good the boxes and arrows are.

• Quality is a design parameter with a number. You are expected to name what a failure is for this system, how it will be measured, and what rate is acceptable. “It should be accurate” is not a requirement.
• Cost per unit is a first-class constraint, in the way latency is in a traditional round. If you do not raise it, the interviewer will, and it is better to have raised it.
• The failure modes are new. Not just “the service is down” but “the service returned something confident and wrong”, which no health check catches.
• Trust boundaries move. Any text that reaches the model is potentially instructions. Where that text comes from a document, a web page or another user, it is untrusted input in a system that has no parser to sanitise it.

One process note before the designs: spend the first three minutes on requirements, and make one of your clarifying questions about consequence. “When this is wrong, who is harmed and how much does it cost?” The answer determines almost every subsequent choice, and asking it early is itself part of what is being scored.

Design 1 — a grounded assistant

“Design an assistant that answers customer questions from our documentation.”

Requirements to pin down first. How large is the corpus and how often does it change? Is the answer shown directly to a customer or suggested to a support agent? What is the worst plausible wrong answer — a mild irritation, or advice with legal or safety consequences? Is there an existing search over this content that already works?

That last question is worth asking because the strongest possible answer to some versions of this prompt is that retrieval plus a summary of the top result is most of the value, and the interviewer usually respects a candidate who checks.

The path. Query → retrieval → rerank → generation with citations → a grounding check → response. The decisions worth speaking aloud:

• Chunking is the quality decision, not the model. Documentation has structure — sections, headings, code blocks — and chunking on that structure beats chunking on a token count. Say that you would keep the heading path in each chunk so a retrieved fragment still knows where it came from.
• Hybrid retrieval, then rerank. Pure vector search loses exact identifiers, error codes and product names, which is most of what people search documentation for. Combining lexical and semantic retrieval then reranking the union is the standard shape, and you should be able to say why each half is there.
• Abstention is a requirement, not a nicety. The system must be able to say the documentation does not cover this. Design the evaluation set to include unanswerable questions from the start, or you will never measure it.
• Freshness. When a page changes, the index must change. Whether that is a webhook, a nightly job or a versioned rebuild depends on how often the docs move and on whether a stale answer is merely wrong or actively harmful.
• Caching. Support questions are heavily repeated, so a cache on near-identical questions can remove a large share of traffic — with the honest caveat that a semantic cache can serve the answer to a subtly different question, so it needs its own correctness story.

The part most candidates skip. How you would know it works: a frozen set of real questions with known-good source passages, measured on retrieval hit rate and on whether the answer is supported by what was retrieved, plus a signal from production such as escalation rate. Say the numbers you would put on a dashboard.

Design 2 — bulk extraction under a budget

“We receive 200,000 supplier documents a month. Turn them into structured records.”

This one is an economics problem wearing a pipeline costume, and the interviewer is watching for whether you notice.

Requirements to pin down. What accuracy is required, and what happens to a record that is wrong — is it caught downstream, or does it become a payment? Is there a deadline per document or only per batch? Are documents scanned images, native text, or both?

The path. Ingest → classify document type → extract to a schema → validate → route to auto-accept or review queue → store, with the review decisions fed back as evaluation data.

• Nothing is user-facing, so batch it. This is the single largest cost lever available and it costs nothing but latency tolerance you already have. Say it early.
• Route by difficulty rather than using one model. A cheap model handles the majority; a cascade escalates only what fails validation. The break-even is computable, and being able to say that the verifier must be much cheaper than the expensive stage — otherwise the cascade cannot win — is a strong moment.
• Validation is deterministic wherever possible. Totals that must add up, dates that must parse, supplier IDs that must exist in a table. These are free, certain, and catch more than a model judge would.
• Confidence routing needs care. A model asked for a confidence score gives a number that is not well calibrated. Prefer structural signals — a missing field, a failed check, a disagreement between two cheap extractions — and be explicit about how the threshold is chosen.
• Idempotency and replay. 200,000 documents means reprocessing will happen. Key the work by a content hash so a rerun is free and a partial failure can resume.

Put a number on it. Estimate tokens per document, multiply out, and state the monthly cost with your assumptions visible. You will be wrong; being wrong with stated assumptions is the skill being tested, and it turns the design into a conversation about which assumption to attack.

Design 3 — an agent with permissions

“Design an assistant that can act in a customer’s account — look up orders, issue refunds, update addresses.”

The security design is the design. A candidate who draws a tool loop and moves on has failed the question regardless of what else they say.

• Name the trifecta. Access to private data, exposure to untrusted content, and the ability to act or communicate externally — the combination is what makes an incident possible, and the design should break at least one leg for any given path.
• Authorisation lives outside the model, always. The tool layer enforces what this user may do, on the server, with the user’s own identity. A prompt instruction is not an access control, and the interviewer wants to hear you say so. Every tool call is an authenticated request.
• Split actions by reversibility. Reads are free. Reversible writes can be automatic with an audit trail. Irreversible or money-moving actions get a confirmation step with a human, and the confirmation must show the actual parameters, not a summary the model wrote.
• Bound the loop. A step cap, a wall-clock deadline and a per-conversation spend ceiling. Without them, one confused conversation is an unbounded bill and an unbounded number of side effects.
• Everything is traced. For an agent, the trace is the audit log: which tools were called, with what arguments, on whose authority. Design it in from the start; it cannot be added after an incident.

The strongest answer to this prompt usually narrows it: start read-only, ship that, and add write actions one at a time behind confirmation as evidence accumulates. Interviewers score “deliver the safe subset first” highly, because it is what the job actually requires.

What the interviewer is scoring

Not the diagram. Roughly, in order of weight:

• Did you establish requirements before designing, including the cost of being wrong?
• Did you say how you would know it works — a measurable definition of failure and a set to measure it on?
• Did you handle the model being wrong, as opposed to the model being down?
• Did you put a number on cost, with visible assumptions?
• Did you name a trade-off and choose, rather than adding every component that exists? Deleting a component under questioning is a positive signal, not a retreat.
• Did you scope down when the problem was too large? Proposing the smallest version that delivers value is the most senior move available in this round.