Model Extraction and Distillation Attacks

“Stealing a model through its API” describes two attacks that share nothing but a name. One recovers parameters. The other recovers behaviour. They have different costs, different feasibility and different defences, and the confusion between them produces a lot of bad advice.

Two different attacks with one name

Extraction of a full frontier model’s weights through an API is not a realistic threat today. Cloning a specific capability of one for a few thousand dollars is routine, and it is what most people who say “model extraction” are actually worried about.

A third attack is regularly filed under the same heading and is genuinely different again: recovering training data rather than the model. Membership inference asks whether a particular record was in the training set, and extraction of memorised sequences has been demonstrated repeatedly against language models — the risk is highest for data that appeared verbatim and rarely, which is precisely the shape of a private document in a fine-tuning corpus. If you fine-tune on customer data, that is the attack to model, and its defences — deduplication, filtering secrets before training, and not fine-tuning on data whose disclosure would matter — have nothing in common with the two above.

What has actually been demonstrated

The academic line starts with Tramèr and colleagues in 2016 (“Stealing Machine Learning Models via Prediction APIs”), which showed exact or near-exact recovery of simple models — logistic regression, decision trees, small networks — from prediction APIs that returned confidence values. The lesson was already the durable one: every extra bit of information in a response is an extra equation for the attacker.

For production language models, the significant result is Carlini and colleagues, “Stealing Part of a Production Language Model” (2024). Working with the vendors and under responsible disclosure, they showed that structural information — the hidden dimension, and the final embedding projection layer — can be recovered through a logit-exposing API for a modest query cost, because the returned logit vectors lie in a subspace whose rank is the hidden dimension. The providers involved changed their APIs in response.

Read that result carefully, because it is frequently overstated. It recovered the final layer and a structural parameter, not the model. What it proved is the general principle: the outputs of an API are a system of equations, and if you expose enough of them you have published the solution.

The API surface that widens it

• Full log-probability vectors. The most information-rich response possible. Returning the top few, rather than a full-vocabulary distribution, is the change that closed the 2024 attack.
• Logit bias. The ability to bias arbitrary tokens lets an attacker interrogate parts of the distribution the API would not otherwise reveal — combined with top-k probabilities, it can be used to reconstruct more than the top-k alone shows.
• Deterministic sampling. Temperature zero makes queries repeatable, which is what a solver needs. Nondeterminism adds noise to every equation.
• Unbounded volume. Both attacks are query-count attacks. Volume is the resource being spent.
• Verbose reasoning traces. For cloning, exposing full chain-of-thought is unusually valuable training material — it is supervision, not just labels.

Defences and what each costs

• Return less. Cap log-probabilities to a small top-k or remove them; restrict logit bias. Cost: legitimate uses exist — classification confidence, evaluation tooling — so scope the capability to keys that need it rather than removing it globally.
• Limit volume, in tokens. Directly raises the price of both attacks. Cost: the usual tension with high-volume legitimate customers, which is why the limit should be per key with an approval path.
• Detect the query distribution. Extraction traffic looks unlike product traffic: systematic coverage, repeated near-identical prompts, log-probability requests, uniform arrival. These are the signals in abuse detection. Cost: false positives on research and evaluation workloads.
• Perturb outputs. Adding noise to returned probabilities degrades the attacker’s equations. Cost: it degrades everyone’s, and a determined attacker averages it away with more queries. Rarely worth it.
• Watermark outputs. Does not prevent cloning; makes it detectable afterwards, which converts a technical problem into an evidentiary one. Statistical watermarking of generated text survives light editing and not heavy paraphrase.

Distillation is mostly a policy problem

Behavioural cloning cannot be prevented by an API design, because the thing being copied is the thing you are selling. If a customer may send prompts and receive good answers, they may keep the answers. Every technical control raises the cost; none changes the shape.

So the controls that matter end up being contractual and evidentiary: terms that prohibit using outputs to train competing models, volume thresholds that trigger a conversation rather than a block, retention of the query logs that would support a claim, and watermarking or canary-style artefacts that make a derived model attributable. Most major providers now prohibit training on outputs in their terms, and the disputes that have arisen have been argued on usage evidence rather than on cryptography.

If you are the one deploying rather than serving models, the same analysis runs in reverse and is worth doing explicitly: your fine-tuned model, exposed through your own API, is subject to exactly this, and the defensible position is to expose the minimum — no log probabilities, bounded volume, and a record of who queried what.