Does AI Actually Make Developers Faster?

This is the question with the highest ratio of confident numbers to citations anywhere in software. Four studies are worth knowing, they disagree, and the disagreement is the finding.

Perception is not evidence

Start with the result that should change how you read everything else. In METR’s 2025 randomised trial, participants using AI assistance took 19% longer to complete their tasks — and afterwards estimated that the assistance had sped them up by roughly 20%. The sign of their own perception was wrong.

That is not a claim that developers are foolish. It is a claim about what the experience feels like: assisted work has less dead time. You are reading and steering rather than typing and looking things up, and the waiting is distributed into small pieces rather than pooled into one visible block. Perceived effort falls even where elapsed time does not.

The consequence is practical. Any measurement of this that rests on self-report — a survey, a satisfaction score, “how much time do you think it saves you” — is measuring perceived effort, and there is direct experimental evidence that perceived effort and elapsed time come apart here. Almost every published productivity figure you will encounter is of that kind.

The four studies, with their designs

Peng, Kalliamvakou, Cihon and Demirer (2023)

“The Impact of AI on Developer Productivity: Evidence from GitHub Copilot”. A randomised controlled trial: 95 freelance developers recruited for the study, randomised into treatment and control, one task — implement an HTTP server in JavaScript. The treated group finished 55.8% faster.

What it supports: on a self-contained, conventional, greenfield task with a supplied test harness, the effect is large and real. What it does not support: anything about maintenance work, about codebases, or about quality — completion time was the outcome, and the study was conducted by researchers at the tool’s vendor, which is disclosed and worth knowing.

Paradis et al. (2024), Google

“How much does AI impact development speed? An enterprise-based randomized controlled trial”. Professional engineers inside a large company, randomised, on enterprise tasks rather than a puzzle. The reported effect was a speed-up, positive but considerably smaller than the freelancer trial. The design is the most representative of ordinary corporate feature work of the four, and it is the least quoted, presumably because a modest number is a bad headline.

METR (2025)

Becker, Rush, Barnes and Rein, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity” (July 2025). 16 experienced open-source maintainers, 246 real issues in their own large, mature repositories, randomised per issue into AI-allowed and AI-disallowed. Result: AI-allowed issues took 19% longer. Beforehand, the developers themselves, plus expert forecasters, predicted a substantial speed-up.

Design limits, which the authors state plainly: 16 participants is small, the repositories are unusually large and unusually well known to the participants, and the tooling is from early 2025. It is evidence about a specific and important regime — expert developers on codebases they know deeply — and not a general refutation.

DORA, State of DevOps (2024 and after)

A large annual survey rather than a trial. It reported that increases in self-reported AI adoption were associated with decreases in delivery throughput and, more markedly, delivery stability. Correlational: adoption was not assigned, respondents self-select, and the outcome measures are also self-reported. It cannot establish causation and does not claim to. Its value is that the association points the same way as METR’s trial, from a completely different method and a much larger sample.

Why they disagree

Line the designs up and the contradiction mostly dissolves. Sort by task rather than by result:

The ordering is exactly the taxonomy on what these tools are good and bad at: context sufficiency, conventionality, verification cost. The effect size is a property of the task, and the studies differ because their tasks differ. Any single percentage quoted without its task description is meaningless, and that includes the two quoted above.

Measuring it in your own organisation

Doable, and almost every attempt gets the design wrong in the same way.

• Randomise by task, not by person. Assigning developers to arms confounds the result with skill, seniority and enthusiasm, and the people who volunteer for the AI arm are not a random sample of anything. Randomise each ticket, in the same team, with the same people.
• Measure elapsed time to merged, on matched tasks. Not lines, not commits, not acceptance rate, not tokens. Lines and commits are outputs, and the whole worry is that outputs rose while outcomes did not.
• Carry a quality outcome alongside it, or you are measuring half the question: change failure rate, and rework — the share of lines modified again within 21 days of being merged. Computing rework from git is a page of its own.
• Pre-register the arm sizes and the analysis. Cycle-time distributions are heavy-tailed, so with 30 tasks per arm the noise will exceed any plausible effect; decide the comparison before you look, and use medians rather than means.
• Never use a survey as the outcome. See the first section. Ask about experience if you like — just do not call it speed.

What can honestly be said

That the effect is real and large on greenfield, conventional, cheaply-verified work; that it is modest on ordinary enterprise feature work; that it can be negative for experts on codebases they know well; that developers cannot perceive which of those regimes they are in; and that nobody has a credible number for “developers” as a population, because the population is not the unit of variation — the task is.

Anything more confident than that paragraph, delivered without a study name and a design, is marketing. That includes figures attributed to consultancies, vendor blog posts reporting internal adoption, and “N% of code is now written by AI”, which measures acceptance of suggestions and not the thing anyone means by it.