Self-Play, and Where It Applies Outside Games

Self-play trains an agent against copies of itself, producing an opponent that is always exactly as strong as the agent is. It has produced some of the field’s most striking results and it transfers badly, for a reason worth being precise about.

Why it works, in three properties

Self-play is not one idea but three, and they happen to coincide in games. Separating them is what tells you whether it can apply anywhere else.

The second property is the famous one and the first is the load-bearing one. Without an exact free outcome, the self-generated opponent produces episodes nobody can score, and the whole loop stops. That is why self-play looks like a general technique and behaves like a special case.

What it achieved, with dates

• TD-Gammon (Tesauro, 1992 onwards). A backgammon network trained largely by playing itself, reaching a level competitive with strong human players, and — the part usually forgotten — producing opening moves that human experts subsequently adopted.
• AlphaGo Zero and AlphaZero (Silver et al., 2017). Trained from random initialisation with no human game records, combining self-play with tree search, and surpassing the earlier version that had learned from human games. AlphaZero applied the same procedure to chess and shogi.
• AlphaStar (Vinyals et al., 2019). StarCraft II, where naive self-play was not enough and the training used a league of diverse agents, including some designed to exploit specific strategies. That detail is the interesting one and the next section explains why.

The failure that is not obvious: cycling

The intuition that self-play produces monotonic improvement assumes strength is transitive: if A beats B and B beats C, then A beats C. Many games are not transitive. Rock-paper-scissors is the trivial example, and real strategy games contain the same structure in more complicated forms.

In a non-transitive game, naive self-play can cycle. The policy learns to beat its current self, which makes it vulnerable to something its older self handled, which it then relearns, indefinitely. Training looks healthy — the agent beats its immediate predecessor every time — while absolute strength goes nowhere.

The mitigations all amount to widening the opponent distribution: play against a pool of past checkpoints rather than only the latest, keep a league of diverse agents, and evaluate against a fixed external reference rather than only against yourself. The diagnostic is straightforward and it is the one to run: play the current policy against checkpoints from several points in training. If it does not reliably beat much older versions, you are cycling rather than improving.

What survives outside games

Most real tasks have none of the three properties. There is no opponent, no rule that decides the outcome, and no unambiguous ordering of policies. The applications that do work take a different route: they construct an adversarial pair where one side produces and the other evaluates, and the evaluation is cheap.

• Generation against verification. One process proposes a solution and another checks it. This is genuine self-play only when checking is meaningfully easier than solving — as it is for a proof, a program with tests, or an equation. Where checking is as hard as solving, the arrangement adds cost and no signal.
• Attack and defence. One model generates attempts to elicit unwanted behaviour and another is trained to resist. The outcome is scored by a classifier, so the reward is not exact and the usual reward-hacking caveats apply, but the self-scaling difficulty property genuinely holds. This is the structure behind automated red-teaming.
• Negotiation and multi-agent simulation. Two agents with opposing objectives generate their own training distribution. The catch is that they optimise against each other and not against reality, and the pair can converge on conventions that work between them and nowhere else — the mechanism behind language drift.

The honest summary of that list is that the second property — a free self-scaling curriculum — transfers reasonably often, and the first — an exact free outcome — almost never does. When someone proposes self-play for a business problem, the question to ask is not about the agents. It is: who decides who won, and what does that cost per episode?

Self-play with language models

Several things called self-play in the language model context are not the same mechanism, and it is worth separating them.

1. Sampling and filtering. Generate many candidate solutions, keep the ones a verifier accepts, fine-tune on those. There is no opponent and no game. It is synthetic data generation with a quality filter, it works well where a verifier exists, and calling it self-play obscures the fact that the verifier is doing all the work.
2. Debate and critique. One model argues, another objects, a judge decides. Genuinely adversarial, but the judge is a model or a human, so the outcome is not free and not exact, and every bias in the judge is now the objective.
3. Prover and verifier training. Train a generator to produce solutions a checker accepts while training the checker to catch wrong solutions. Closest in structure to real self-play, and it depends entirely on the checker being genuinely harder to fool than the generator is to improve. When that fails, the pair reaches an equilibrium where the generator produces things the checker happens to like and neither is any good.

The first is used constantly and works. The third is a live research direction. The middle one is where most of the enthusiasm sits and where the least is established.

A test for whether your problem qualifies

Four questions, in the order that fails fastest.

1. Can you score an episode automatically, exactly and cheaply? If no, stop. Everything else depends on this and no amount of architecture substitutes for it.
2. Is there a second party whose objective genuinely opposes? Not a critic you invented, but a real opposing objective. Without one you have sampling and filtering, which is fine and is not self-play.
3. Is strength transitive enough? Test by having the current policy face old checkpoints. If it does not dominate them, expect cycling and plan a league.
4. Does winning against yourself correspond to being good at the real task? Two agents can co-evolve into a joint strategy that is excellent against each other and useless against anything else. A fixed external benchmark is the only thing that detects this, and it has to be external.

Most business problems fail the first question, and it is worth failing it quickly. The good news is that failing it points somewhere specific: if you cannot score an episode, the project is a reward modelling problem or a verifier problem, and solving that is a prerequisite for any RL method at all, not just this one.