AI Coding Assistants: What They're Good and Bad At
5 min read · updated August 3, 2026
“Is it good at Python?” is the wrong question, and it is the one every comparison answers. The property that decides whether a model helps is not the language. It is how much of the correct answer is visible in the context you can afford to give it.
The axis everyone compares on is wrong
Comparisons of coding assistants are organised by language, framework and IDE. Those axes barely matter. A model that writes competent Rust writes competent Python; a model that fails on your service will usually fail on it in any language, because the thing it is missing is not syntax. The variation that matters is between tasks, and it is enormous — large enough that the same tool, measured properly, produces a substantial speed-up on one task type and a slowdown on another.
The mechanism explains why. A model returns a likely continuation of its context. So the question “will this work” decomposes into: is the answer derivable from what I can show it, is the conventional answer the correct one, and can I check the result cheaply enough that being wrong is not expensive.
Three properties that actually predict it
| Property | Description |
|---|---|
| context sufficiency | Is everything needed to be correct present in the tokens you can afford to send? A pure function with its types and a docstring: yes. A change that must respect an invariant enforced three services away and written down nowhere: no. This is the dominant term. |
| conventionality | Is the correct solution also the most common one? Parsing a CSV: yes. Anything security-sensitive: no — the common pattern on the public internet is frequently the vulnerable one, and the model's prior is that corpus. |
| verification cost | What does it cost to find out you were wrong? A failing unit test in two seconds is cheap. A subtle behaviour change discovered in production three weeks later is not. Low verification cost turns a mediocre suggestion into a cheap draft; high verification cost turns it into a liability. |
These are not independent of each other in practice — mature codebases tend to score low on all three at once, and greenfield scripts tend to score high on all three — but they fail differently, and knowing which one is low tells you what to fix. Low context sufficiency is fixable by assembling better context. Low verification cost is fixable by writing the test first. Low conventionality is not fixable at all, and is the signal to do it yourself.
Where each class of task lands
High on all three — delegate freely
- Mechanical translation between formats. JSON to a typed struct, an OpenAPI fragment to a client method, a SQL schema to migration boilerplate. The answer is fully determined by the input and the compiler checks it.
- Incantations you would otherwise look up. A regex, a
jqfilter, anffmpeginvocation, a window function. Conventional by construction, and verifiable in one run. - Test scaffolding and fixtures where the shape is obvious and the cases are enumerable.
- First drafts of self-contained scripts. The classic demo case, and the one the published trials that show large speed-ups actually measured.
Low on context sufficiency — the expensive middle
- A change that must be consistent across many call sites. The model edits what it sees; it cannot know about the fourth caller unless the fourth caller is in the context. This is the mechanism behind duplicated helpers and the source of the duplication that accumulates.
- Anything depending on an unwritten invariant. “Never call this outside a transaction”, “this queue is at-least-once so handlers must be idempotent”. Write it in the repo instruction file and the task moves up a band.
- Dependency and version work. The model has seen many versions of a library and does not reliably condition on the one in your lockfile.
Low on conventionality or verification cost — do it yourself
- Authorization boundaries. The happy path is what is predictable; the ownership check is exactly the code that is absent from most public examples. See the classes of security bug this produces.
- Concurrency, locking and transaction boundaries. Verification is expensive because failure is intermittent, and the conventional answer is frequently subtly wrong.
- Performance work where the bottleneck is the data. The model optimises the code it can see; the cost is usually in a query plan or an access pattern it cannot.
Why two trials disagree so violently
Two randomised controlled trials are usually cited on opposite sides of this argument, and read through the taxonomy they are not in conflict at all.
Peng, Kalliamvakou, Cihon and Demirer (2023) recruited 95 freelance developers and randomised access to GitHub Copilot on a single task: implement an HTTP server in JavaScript. The treated group finished 55.8% faster. That task is high on all three properties — greenfield, entirely conventional, and verified by a supplied test harness in seconds.
METR’s 2025 trial (Becker, Rush, Barnes and Rein, July 2025) randomised AI assistance across 246 real issues in the mature open-source repositories that its 16 participants already maintained. Issues where AI was allowed took 19% longer. Those tasks are low on all three: the invariants live in the maintainers’ heads, the correct fix is often the unconventional one that fits the existing design, and verification means understanding a codebase the participant knows better than any context window could convey.
Neither result generalises to “AI helps” or “AI does not help”. Both are consistent with the effect being a property of the task. The full reading of the literature, including the studies that sit between these two, is on the productivity page.
Scoring your own backlog
Score each of the three properties 0, 1 or 2 for a ticket before you start. Context sufficiency: 2 if you can name every file that must change and they fit in the window; 0 if you cannot. Conventionality: 2 if a competent stranger would write the same thing; 0 if the right answer is specific to your system. Verification: 2 if a test proves it in under a minute; 0 if you would find out in production.
Five or six: delegate the whole thing and review the diff. Three or four: use the model for the plan and the boilerplate, write the load-bearing part yourself. Two or below: the model is a rubber duck, and the time it costs to correct it will exceed the time it saves. The rubric takes fifteen seconds and its only real function is to stop you from spending forty minutes discovering the score empirically.