Training Your Team to Use AI Well

Most internal AI training is a tour of a product interface followed by a list of prompt tips. Both age out within a release, and neither addresses the thing that actually goes wrong: people do not know when to believe the output.

What training can and cannot change

Be honest about the scope. A workshop cannot make anyone better at the underlying job, cannot teach a durable set of prompt tricks — the tricks change with each model generation — and cannot substitute for a tool that does not work.

What it can change is calibration: the accuracy of a person’s judgement about when the output is likely to be right, and what to do when they are unsure. That is learnable, it transfers across tools, and it survives model releases, because it comes from understanding one mechanical fact — the model is producing a likely continuation, and there is no truth term anywhere in that process. Every exercise below exists to install that fact experientially rather than as a slide.

The second thing it can change is knowledge of the boundary: what may be pasted into what. That is not a matter of judgement, it is a matter of a rule people can recall, and the recall is testable.

The agenda

Three hours, one room, everyone with a laptop and access to the actual tool they will use. Not a recording — the exercises need people to be surprised at the same time.

0:00  What it is                          20 min
      One idea: it predicts a likely continuation. No training
      history, no architecture, no benchmarks. Then the four
      consequences: it can be fluent and wrong; it knows only
      what is in the request; it cannot check arithmetic or
      dates reliably; it does not remember yesterday.

0:20  EXERCISE 1 — break it on purpose    30 min
0:50  EXERCISE 2 — the verification drill 30 min
1:20  break                               10 min
1:30  EXERCISE 3 — context beats cleverness 30 min
2:00  EXERCISE 4 — the data line          30 min
2:30  Your team's actual work              25 min
      Three real tasks from this team, done live, with the
      room deciding which are suitable and why.
2:55  Where to ask questions               5 min

Note the ratio: two hours of doing, forty minutes of talking. The talk exists to give the doing a vocabulary, not the other way round.

The four exercises

1 — Break it on purpose

Each person spends twenty minutes trying to get the model to state, confidently and in detail, something false about their own area of expertise. Ask for a specific internal policy, an obscure regulation, a product limit, a citation. Then share the best three with the room.

The purpose is emotional rather than informational. Reading that models can be confidently wrong changes nobody’s behaviour; watching one invent a plausible clause number in your own field changes it permanently. It also sets up the vocabulary for the rest of the day — particularly fabricated references, which is the failure most likely to reach a customer intact because it looks exactly like the real thing.

2 — The verification drill

Take three real outputs from exercise 1 or from the tool’s normal use. For each, the group writes down the specific check that would catch a mistake, and how long it takes. Not “check it carefully” — the actual check: open the policy document and search for the clause; recompute the total; look the customer up in the CRM.

The output of this exercise is a short list of checks that are cheap relative to the task. That list is the real deliverable of the whole workshop, and it is team-specific. It also surfaces the tasks where no cheap check exists — which are the tasks this team should not be using the tool for yet, and knowing which those are is worth the three hours on its own.

3 — Context beats cleverness

Everyone runs the same task twice: once with a clever instruction and no source material, once with a plain instruction and the relevant document pasted in. The second wins, visibly, almost every time.

This is the antidote to prompt superstition, and it is the single highest-leverage habit to install, because it generalises: the model can only work with what is in the request. It is also the everyday version of grounding as an architecture decision, which is worth naming so that people recognise the same idea when it shows up in a build discussion.

4 — The data line

Twelve slips of paper, each a real-shaped input: a customer email with a name in it, a draft contract, a spreadsheet of salaries, an internal memo, a public press release, a support ticket with an account number, a CV, a screenshot of a dashboard. Groups sort them into three piles: fine, fine after removing something, not into this tool.

Then reveal the intended answer and argue about the disagreements. The arguments are the point: they show exactly where the policy is ambiguous, which is information you cannot get any other way, and the policy should be edited afterwards to close whatever the room disagreed about.

What not to teach

• Prompt formulas. “Act as a senior…” and the rest. They are folklore, they change between model generations, and teaching them wastes the hour that exercise 3 needs.
• Model names and benchmarks. Nobody choosing between two models in a meeting is helped by a leaderboard, and staff using an approved tool are not choosing at all.
• Token and pricing trivia. Relevant to whoever owns the budget; irrelevant to the person writing a reply, and it makes the session feel like it is about the vendor.
• A tour of the interface. Fifteen minutes of the product’s own documentation, sent in advance. The room is for things that need a room.
• Injection defence as a user responsibility. Users should know that content from outside can carry instructions — the indirect case is the one that matters — but the defence is architectural, and teaching people to spot it is teaching them to do something they cannot reliably do.

Telling whether it worked

Not a satisfaction score. Satisfaction with a workshop measures the workshop; you want to know whether behaviour changed, and three proxies are available without instrumenting anyone.

• Edit rate. If the tool produces drafts, the fraction of drafts edited before use is a direct measure of whether people are reviewing. A very low edit rate immediately after training is a warning, not a success.
• The shape of the questions. Before training, questions are about how to use the tool. After, they should be about whether a specific task is appropriate for it. That shift is visible in a support channel without any measurement apparatus.
• Policy incidents. The data-line exercise should reduce the rate of the boring boundary mistakes. It will not reduce it to zero, and the residual is useful: each one names a case the policy did not cover.

Repeat the session for new joiners on a fixed cadence rather than as a one-off event, and refresh exercise 4 whenever the policy changes. The rest of the material ages slowly, because it is about a property of the technology rather than about a version of a product.