The Tasks Where AI Is the Wrong Choice

The useful question is not what the technology can do. It is where the cost of an undetected error dwarfs the time saved, because that is the only place where a tool that is right most of the time is the wrong tool.

The test

Three properties. A task with all three belongs on the list, and the list below is simply the common cases.

1. The error is hard to see. Fluent, confident and structurally correct output that happens to be wrong produces no signal. Compare a wrong bus timetable, which you notice at the bus stop.
2. The consequence is not reversible. A sent message, a filed return, a decision communicated to a person, a dose administered. Anything you can redo cheaply is not on this list.
3. Checking costs as much as doing. If verifying the output requires the same expertise and time as producing it, the saving is zero and you have added a source of error for nothing.

Two out of three is a caution. Three out of three is a no. And notice that none of the three is about capability — a better model changes the frequency of errors and changes none of the three properties.

The list

The reasoning behind several of these is the same: the domains where a confident wrong answer is hardest to detect are the domains with the highest cost of being wrong, which is an unhappy correlation but a reliable one.

Why a better model does not move the line

The standard objection to any list like this is that it will be obsolete when the next model arrives. It will not, and the reason is arithmetic rather than scepticism.

A lower error rate reduces how often you are wrong. It does nothing to the other two properties in the test: the errors remain invisible, because they are produced by the same process that produces the correct answers, and the consequence remains irreversible. Work through what a very good error rate actually implies for a task you do often.

ASSUMPTIONS (illustrative)
  task done                        3 times a week
  weeks                            50
  uses per year                    150

  error rate 5%   ->  7.5 wrong answers a year
  error rate 1%   ->  1.5 wrong answers a year
  error rate 0.1% ->  0.15, or one every seven years

WHAT CHANGES AT EACH STEP
  The number of errors falls by a factor of ten each time.
  Which ones are wrong: still unknown without checking.
  Your ability to notice one: unchanged.
  Cost of the one that gets through: unchanged.

AND THE PART THAT GETS WORSE
  At 5% you check, because you have been burned.
  At 0.1% you stop checking, because it is always right.
  The one error in seven years arrives in a year when nobody
  in the organisation has ever seen it be wrong.

That last effect is well documented in other automated systems, where it is usually called automation bias: reliability produces trust, trust removes the check, and the rare failure then propagates further than it would have when the system was worse. It is why aviation and medicine build the verification step into the procedure rather than leaving it to judgement.

So improvement moves tasks off this list only when it changes detectability or reversibility, not when it changes accuracy. A tool that showed its working in a form a non-expert could check would move several of these. A tool that is merely right more often does not move any of them.

Near misses: tasks that look like they belong

A list of prohibitions is only credible if it is short, so it is worth naming the tasks that fail the test and are commonly treated as though they passed it.

• First drafts of important documents. The error is visible — you read it — and it is entirely reversible. An important document is a reason to review carefully, not a reason to face a blank page.
• Anything creative. A bad idea costs nothing. The objections here are about attribution and craft rather than risk, and they are real, but they are a different argument.
• Learning a subject. Errors are correctable, checking is cheap against a textbook, and the alternative is often not learning it. The methods that work are in AI for studying.
• Summarising for triage. Deciding what to read is low-stakes; deciding what a document obliges you to do is not. The distinction is which decision the summary supports.
• Code. Frequently wrong, and the wrongness is detectable by running it and by review, which is why the software industry adopted this faster than anyone. Test coverage is the checking mechanism the other domains lack.

The grey zone

Some tasks move across the line depending on how they are set up, and these are the interesting ones.

Anything with numbers is fine when the arithmetic is done by something that does arithmetic, and not fine when a number appears in a sentence because the sentence needed one. Move totals into a spreadsheet and the task changes category, which is why spreadsheet work is safe with a verification step and unsafe without one.

Anything with sources is fine when every claim is quoted from a document you supplied and checkable in seconds, and not fine when the sources come from the model. The same task, two arrangements, opposite risk profiles.

Anything about a person is fine when it drafts a communication you decided on, and not fine when it supplies the decision. Drafting the letter is admin. Choosing who gets it is not.

Anything a professional signs is fine as preparatory work that reduces their time, and not fine as a replacement for their judgement. The pattern across this whole section: the tool may do the part that is cheap to check.

Saying no when somebody asks you to

The list is easy to agree with and harder to apply when a manager asks you to shortcut something on it. Three framings work better than a general objection, because each converts a disagreement about technology into a question about process, which is a conversation organisations know how to have.

• Name the check rather than refusing the tool. “I can draft it that way; it will need a named reviewer before it goes out, and that will take an hour.” This is almost always accepted, and it puts the real cost where it belongs instead of leaving it hidden.
• Ask who signs it. For anything attested, filed or sent to a regulator, the question “whose name is on this?” settles the matter without an argument about capability. The person whose name is on it has to have read it.
• Offer the version that is safe. Almost every task on the list has an adjacent one that is not: not the diagnosis but the list of questions for the clinician; not the legal advice but the summary of where to look; not the decision about a person but the letter conveying a decision you made.

Where none of that lands, put your position in writing once, briefly and without drama. That is not an act of defiance; it is how a reasonable objection becomes part of the record, and it is what protects you if the risk you identified is the one that materialises.

When you have to anyway

Sometimes the alternative is nothing at all: no lawyer, no translator, no clinician, no time. That is a real situation and pretending otherwise helps nobody. Four rules make it as safe as it can be.

1. Write down that you did it. Note which parts came from a model and were not verified. The failure that causes damage is the one where nobody afterwards can tell which parts were checked.
2. Verify the checkable parts, all of them. Every name, date, figure, citation and reference. This is where errors concentrate, and it is affordable — the routine takes five minutes.
3. Get one human look, even a cheap one. A half-hour consultation, a colleague who knows the area, a helpline. A single informed read catches the class of error that verification cannot, which is the consideration that never came up.
4. Delay anything irreversible. If the decision can wait a week for a proper answer, it should. Urgency is frequently manufactured, and the asymmetry that put the task on this list does not go away because you are in a hurry.