AI Detection in Schools: The Harm of False Positives

A detector advertised as highly accurate can be both accurate as described and disastrous in use. The gap is not a flaw in the product claim; it is what happens when a classifier meets a population where most cases are negative, and it is calculable in advance from the vendor’s own published figure.

The arithmetic, with the vendor’s own number

Substitute whatever false-positive rate a tool advertises. The structure does not change.

Illustrative. Substitute your own institution's numbers.

  submissions this term            5,000
  actually AI-written                500   (10%)
  detector catches                    80% of those
  detector false-positive rate         1%   (the advertised figure)

  true flags     500 x 0.80          = 400
  false flags  4,500 x 0.01          =  45

  445 accusations, 45 of them wrong — one in ten
  and 100 AI-written submissions passed unflagged

Forty-five students accused wrongly, from a tool whose error rate is one percent. Raise the false-positive rate to 3%, which is still a respectable-sounding figure, and it becomes 135 wrongly accused against 400 correctly — one wrong accusation for every three right ones. Lower the base rate, which happens in any cohort where cheating is less common, and the ratio gets worse still, because the false positives are drawn from the larger population.

This is base-rate neglect in a setting where the cost of an error falls entirely on one person. No improvement in catching cheats fixes it: the false positives come from the size of the innocent population, not from the detector being bad at its job.

What the detector is actually measuring

Text detectors do not identify a signature left by a model. There is no such signature to find in ordinary text. They estimate statistical properties of the writing and compare them to what model-generated text tends to look like — broadly, how predictable each word is given the preceding ones, and how much that predictability varies across the document. Model output tends to sit in a narrower, more predictable band than human writing, because sampling concentrates on high-probability continuations.

That is a real regularity and it is also a description of a writing style rather than of an author. Any human writing that is conventional, plain and structurally regular occupies the same region, and so the detector is measuring conventionality with model authorship as a proxy. How text detection works in more detail covers the mechanics, and watermarking covers the only approach that could in principle carry a real signal — and its limits.

Two consequences follow directly from the mechanism. Light editing of model output moves it out of the detected region, so the tool is weakest against exactly the students who put effort into evading it. And a student who writes plainly is in the detected region without having done anything.

Why the errors are not evenly spread

If the detector responds to plain, conventional, low-variance prose, then the false positives concentrate on people who write that way, and that is not a random sample of students.

• Writers working in a second language tend toward simpler constructions and a more limited vocabulary — a deliberate, skilful strategy that produces exactly the statistical profile the detector responds to.
• Students taught to write to a formula — a prescribed essay structure, a required register — are producing low-variance text on instruction.
• Technical and formal registers are constrained by convention. Lab reports and legal writing are supposed to be predictable.
• Students using ordinary assistive tools — grammar checkers, style suggestions — have text that has been smoothed toward the conventional, which is the same direction.

So the burden of false accusation lands on students who are already more likely to be doubted and less likely to be believed when they appeal. That is a foreseeable consequence of the mechanism rather than an unlucky outcome.

The asymmetry of the two errors

Any classifier makes two kinds of mistake, and institutions deploying these tools frequently treat them as comparable. They are not. A missed case of cheating costs one grade that was slightly too high. A false accusation puts a student through a disciplinary process, against evidence they cannot rebut, over an allegation about their own mental work.

The rebuttal problem is the sharpest part. Asked to prove they wrote it, a student has no positive evidence available — the absence of model use leaves no trace. Version history helps if it exists, and most people do not draft in a system that keeps one. An accusation from a probabilistic classifier is therefore close to unfalsifiable by the accused, which is a property no evidence in a disciplinary process should have.

What works instead

• Never use a score as evidence on its own. A flag is a reason to look, and looking means reading the work and talking to the student. If the score is what convicts, the arithmetic above is the conviction rate.
• Assess the process, not only the artefact. Drafts, outlines, in-class writing and a short conversation about the argument are all evidence a detector cannot produce and a model cannot supply.
• Design assessments that model use does not defeat. Work tied to material only the student has — a specific seminar discussion, their own data, a critique of their earlier draft — changes the incentive rather than policing it. See what changes in assessment.
• State the policy before the term, not after a flag. Most disputes are about a rule that was never written down.