Detecting AI-Written Text: Why Detectors Fail

A detector with an impressive accuracy figure can still produce more false accusations than true ones. That is not a criticism of any particular product; it follows from the base rate, and it is the first thing anyone deploying one should compute.

How detectors work

Three families, with different failure profiles:

• Statistical signatures. Model-generated text tends to sit in a region of low perplexity and low burstiness under a reference model — it is, by construction, made of likely continuations. Methods in the DetectGPT family go further and look at the curvature of the log-likelihood around the text, on the reasoning that generated text sits near a local maximum. Principled, and dependent on a reference model that may not match the generator.
• Trained classifiers. Fine-tune a model on human and machine text and let it learn the distinction. Performs well on the distribution it was trained on and degrades on new generators, new domains, new prompting styles and different languages — the standard distribution-shift problem.
• Watermark detectors. Categorically different: they test for a signal a cooperating generator deliberately inserted. Much stronger when applicable, and only applicable to that generator, with the limits covered on the watermarking page.

The arithmetic that decides everything

Any detector has an operating point: a true positive rate and a false positive rate, which move together as you slide the threshold. Applied to a population, what matters is not accuracy but precision — of the items it flags, what fraction are actually machine-written.

Let  N    = number of submissions
     p    = prevalence of machine-written work, as a fraction
     TPR  = true positive rate at the chosen threshold
     FPR  = false positive rate at the same threshold

  true positives   =  N · p · TPR
  false positives  =  N · (1 - p) · FPR

  precision  =        p · TPR
               ---------------------------
               p · TPR  +  (1 - p) · FPR

  false accusations per true catch  =  (1 - p) · FPR
                                       ----------------
                                            p · TPR

Put a vendor’s own claimed numbers into that expression with a prevalence you believe. Two properties emerge that no improvement in the detector removes. First, when p is small the (1 - p) · FPR term dominates, and precision collapses however good the true positive rate is — the classic base rate problem. Second, precision is a property of the population, not of the tool, so a figure quoted from a balanced benchmark where half the samples were machine-written tells you nothing about a cohort where a small minority is.

Then weight the two errors, because they are not comparable. A missed detection costs an unearned grade. A false positive costs a person an accusation of dishonesty they cannot disprove. If you regard the second as many times worse, the threshold has to move far toward specificity, and at that setting the detector catches very little — which is the honest conclusion the arithmetic points at.

The errors are not random

Liang and colleagues reported in 2023 that several detectors classified writing by non-native English speakers as machine-generated at substantially higher rates than writing by native speakers, and that simple prompting to enrich the language reduced the misclassification — which is itself diagnostic of what the detectors are keying on.

The mechanism is not mysterious and it generalises beyond that study. Detectors respond to low perplexity and limited lexical variation. Competent second-language writing is often more conventional in construction and more restricted in vocabulary, for reasons that have nothing to do with authorship. So do formulaic disciplinary genres — lab reports, legal memos, technical documentation — and so does text that has been through a grammar checker or a translation tool. Autistic writers and writers taught to a rigid template have both been raised as affected populations, though with less published measurement behind the claim.

The consequence is the important part: the false positives concentrate. A tool with a low average false positive rate that is elevated for one group produces a pattern of accusations that tracks that group, which is a discrimination problem with an ordinary statistical explanation rather than a rare accident.

The incidence is inverted

Consider who the detector actually catches. Someone determined to evade it paraphrases, asks for a different register, edits by hand, or runs the output through another model. All of these are trivial and all of them work, as the watermarking attack literature and the distribution-shift results both indicate.

What is left is the careless user and, through the mechanism above, the honest writer whose prose happens to look statistically machine-like. So the population flagged skews toward the least sophisticated offender and the innocent, and away from the deliberate one. Any enforcement regime built on a detector is therefore, structurally, a regime that penalises unsophistication — including unsophistication in evading a tool that nobody was told was being applied.

A defensible use, if there is one

• Never as evidence. A score from a proprietary model is not something the accused can examine, rebut or reproduce. A process that treats it as proof denies the basic ability to contest.
• Never automated. No sanction, grade penalty or account action triggered by a score alone.
• At most, a prompt to look. A flag can justify reading the work properly, asking about the argument, or checking process evidence — and the conversation, not the score, is then the basis for any conclusion.
• Disclose it. Tell people a detector is in use, which one, and what its known error profile is. An institution unwilling to publish that is admitting the score would not survive scrutiny.
• Set the threshold with the arithmetic above. If the institution will not compute the expected number of false accusations in a cohort before deploying, it has not made a decision — it has accepted a vendor’s default.
• Prefer process evidence. Drafts, revision history and a five-minute conversation about the work are more probative, available, and rebuttable than any classifier output, and they do not single out anyone’s writing style.

Detector performance and the published evidence both change; re-check before relying on any of it, including this page.