AI Engineering Interview Questions, and What a Strong Answer Contains

A list of interview questions is only useful if it tells you what is being tested. Otherwise it produces candidates who have memorised the answer to a question the interviewer was using as an opening, and who fall over on the second sentence.

Where these come from

These questions are constructed from the work rather than collected from interviews — nobody here surveyed hiring managers, and a page claiming to have done so would be inventing its sources. What makes them representative is that each maps onto a decision the job actually contains, which is the same constraint a competent interviewer works under when writing their own.

Read them for the structure of a good answer rather than the content. The pattern that recurs: a strong answer names the trade-off, gives a concrete example, and states the condition under which the answer would change. A weak answer is a definition.

Fundamentals

1. Why does the same prompt sometimes give a different answer?

Testing: whether you know that the model returns a distribution and something outside it picks a token. This is the question that separates people who have used an API from people who understand what is behind it.

A strong answer covers the sampler and the parameters that control it, and then goes one step further: temperature zero is not a guarantee of determinism in a real serving stack, because batching and floating-point non-associativity can change the result. Mentioning that unprompted is a strong signal.

Follow-up: “So how would you write a test for a feature built on this?” The memorised answer stops at temperature zero. The real one talks about asserting on properties rather than exact strings, and about running enough samples to measure a rate.

2. What is a token, and why should I care about the difference between input and output?

Testing: whether the economics of the system are real to you.

A strong answer connects tokenisation to three separate consequences — price asymmetry, latency (prefill is parallel, decoding is not), and context limits — and mentions that code, JSON and non-Latin scripts tokenise worse than English prose, so estimates from word counts mislead.

Follow-up: “Your bill tripled last month and traffic is flat. Where do you look?” Good answers go straight to prompt growth, conversation history, retrieved context, and reasoning tokens you are billed for but never see.

3. When would you not use RAG?

Testing: whether you reach for a pattern reflexively.

A strong answer names the conditions: the corpus is small enough to fit in context, the task needs the whole document rather than passages, the knowledge is stable and better placed in the system prompt, or the real problem is behaviour rather than knowledge — in which case the retrieval-versus-tuning distinction applies. A very strong answer notes that retrieval adds a failure mode that did not exist before.

Systems and failure

4. Your provider starts returning 429s at 10% of requests. What happens to your service?

Testing: production instincts. This one is diagnostic; the answers separate cleanly.

A strong answer distinguishes retryable from non-retryable, describes backoff with jitter and a budget, notes that naive retries make the overload worse, and considers shedding load or degrading rather than queueing indefinitely. It also asks whether the work is user-facing or batch, because the answer is different.

Follow-up: “You add a fallback to a second provider. What have you just made worse?” The candidate should reach output differences, cost differences, and the fact that the fallback path is the least-tested code in the system.

5. How do you know a change to a prompt was an improvement?

Testing: whether you can distinguish evidence from impression. Probably the single highest-signal question on this list.

A strong answer has a frozen set with a written failure definition, an idea of how many examples are needed to see a difference of the size that matters, and awareness that a model judge has biases that need controlling. The best answers admit the uncomfortable part: most prompt changes are not measurable improvements, and knowing that is what stops a team churning.

6. A user reports the assistant made something up. Walk me through the debug.

Testing: whether you localise before you fix.

A strong answer is a bisection: was the information in the retrieved context at all (retrieval failure), was it in context and ignored (generation failure), was it in context and contradicted (grounding failure), or was the question outside the corpus and the system failed to abstain? Each has a different fix, and the distinction between them is the whole diagnosis. Answers that begin “I would tell it not to hallucinate” are the failure mode this question exists to catch.

Judgement and trade-offs

7. When would you choose a smaller, cheaper model?

Testing: whether “good enough” is a number to you or a feeling.

A strong answer makes it a comparison between the money saved and the extra failures caused, valued at what a failure costs — which turns into a break-even failure cost you can compute. It also notes that if failures are caught by a validator or a review queue, the cost of a failure drops sharply and the answer changes.

8. When is an agent the wrong tool?

Testing: resistance to fashion.

A strong answer points out that an agent is a loop with unbounded cost and latency, that a fixed pipeline is testable in ways a loop is not, and that most tasks framed as agentic are a chain of three known steps. It should mention that stopping conditions are the hardest part and that an agent with real permissions is a security design problem, not just an engineering one.

The behavioural one that matters

9. Tell me about something you built with a model that did not work.

Testing: honesty, and whether you learn from evidence. Every candidate has one; the ones who claim otherwise have not shipped.

A strong answer is specific about what the failure was, how it was noticed (ideally by a measurement rather than by a complaint), what the diagnosis turned out to be, and what changed afterwards — in the system or in the process. The strongest versions include a wrong hypothesis the candidate held for a while, because that is what real debugging contains.

Follow-up: “What would you have needed to catch it a week earlier?” This is where logging and traces come up naturally, and it is a far better way to reach that topic than asking about observability directly.

What to ask them

Interviews are two-way, and in this field a few questions reveal almost everything about whether the job is real work or a demo treadmill.

• “How do you currently know if a change made the system better?” If the answer is a person trying it, the team is early — which may be fine, but you should know.
• “What happens when a provider ships a model update you did not ask for?” A team that has thought about silent updates has operated something real.
• “What is in production today, and how long has it been there?” The gap between the demo and the deployed thing is the most useful number you can extract from an interview.
• “Who owns the AI budget?” A shrug means cost work will eventually land on you without authority to do anything about it.