Autocomplete vs Chat vs Agent: Three Different Tools

These are not three interfaces to one product. They are three designs with incompatible constraints, and almost everything people dislike about each one is that constraint showing through.

The constraint that separates them

Autocomplete competes with your own typing. If a suggestion arrives after you have already typed the line, it is not a suggestion, it is an interruption — which puts the whole round trip somewhere in the low hundreds of milliseconds. That budget dictates everything downstream: a small model, a short context, no reasoning tokens, and no opportunity to look anything up.

Chat has a budget of a few seconds because you are waiting on purpose. An agent has a budget of minutes because you have gone to do something else. Each order of magnitude buys a qualitatively different amount of work — and buys it at a proportionally different price, which is why the three meters look nothing alike.

Autocomplete: fill in the middle

The request is not “continue this text”. Your cursor has code after it as well as before it, so the model is asked to infill. That is a distinct training objective — Bavarian et al., “Efficient Training of Language Models to Fill in the Middle” (2022) — and it shows up as literal sentinel tokens in the request:

<|fim_prefix|>function parseDuration(s: string): number {
  const m = /^(\d+)(ms|s|m|h)$/.exec(s);
  if (!m) throw new Error("bad duration: " + s);
<|fim_suffix|>
}
<|fim_middle|>

Code Llama uses <PRE>, <SUF> and <MID> for the same three roles. A base model without this objective, asked to autocomplete, will happily rewrite the suffix or re-emit the closing brace, because nothing told it there was a suffix.

The failure mode follows from the budget: the context is a window around the cursor plus, at best, a few recently-open files. So autocomplete invents API surface — a method name that is a perfectly plausible continuation of your object and does not exist. It is right often enough that you stop reading, which is precisely when it costs you. Treat an accepted completion as typed by a stranger: your compiler and your tests are the review, not your eyes.

One more thing about this mode, because it is the number everyone reaches for: acceptance rate — the share of suggestions accepted — is not a quality measure and should not be treated as one. It rises when suggestions are short and conventional, which is when they are least valuable, and it counts a tab press that you immediately typed over as a success. It is a usage statistic. Reporting it as “the share of our code written by AI” is the single most common category error in this whole subject.

Chat: you are the retrieval system

In chat the model gets exactly what you pasted or attached, and nothing else. That is a feature — you control the context precisely, which is the highest-leverage variable there is — and it is the whole failure mode, because you will forget something and the model has no way to notice, no way to ask for the file it needs, and every incentive to answer anyway.

The second failure mode is the edit format. Chat returns prose containing code, so you apply it, and the standard disaster is the “here is the updated file” response that silently drops a function it was not shown. Ask for a unified diff or for before/after blocks rather than a whole file, and the omission becomes visible instead of silent.

Chat is the right mode when the unit of work is understanding rather than editing: what does this stack trace mean, why is this query slow, what are the three ways to structure this. It is the wrong mode for a change spanning more than about two files, because you become the bottleneck twice — once assembling context and once applying edits.

Agent: it retrieves, you own the sandbox

An agent gets tools — read a file, search, apply an edit, run a command — and loops until something stops it. The context problem becomes the model’s: it can grep for the caller it needs, which is a genuine improvement over one-shot retrieval, at the cost of turns and tokens.

What you own instead are the two things it cannot own. The sandbox, because a loop with shell access and no network policy will eventually install a package to make an error go away. And the stopping condition, because a model has no notion of “this is not working” and will iterate confidently past the point of value. Both are designed rather than configured.

The failure mode is drift: twenty minutes later there is a 400-line diff, the test is green, and three of the changes are unrelated to the task. The mitigation is the same one that works for humans — small scopes and frequent commits — plus a hard rule that a diff you have not read is not merged, however green it is.

Choosing per task