Four Cheaper Methods That Beat RL on Most Real Problems

Reinforcement learning is the right tool for a narrow class of problems: sequential decisions, where the action changes what happens next, where you can score outcomes but cannot demonstrate good behaviour. Most problems brought to it are not in that class, and four cheaper methods cover nearly all of the rest.

The position, stated plainly

RL buys one thing: the ability to optimise a behaviour you can score but cannot demonstrate. Everything else about it is a cost. If you can demonstrate the behaviour, imitate it. If the decision does not affect the next decision, use a bandit. If you have a model of the system, use a solver. If three rules would do it, write three rules.

The reason this needs saying is that RL is the most interesting method on the list, and the search results are written by the people whose projects worked rather than by the larger number who tried and reverted. What follows is the triage, in the order it should be attempted.

1. Supervised learning on logged decisions

When it applies: you have records of decisions and how they turned out, and some of those decisions were good.

Filter the logs to the outcomes you would want repeated and train a model to reproduce those decisions. This is behaviour cloning, and it is the highest-value first attempt on almost any decision problem, because it needs no reward function, no environment, no exploration and no stability engineering. It is a supervised learning problem with an ordinary loss and an ordinary validation set.

For language models the equivalent is supervised fine-tuning on filtered good outputs — generate several candidates, keep the ones that pass a check, train on those. It captures a large share of what an RL run would deliver at a fraction of the complexity, and it is what most teams should try before anything else in this cluster.

Its ceiling: imitation cannot exceed the best behaviour in the data, and it inherits every bias in how the data was collected. When you have genuinely hit that ceiling — measured, not assumed — a preference method is the next step and full RL is the one after that.

2. Bandits

When it applies: the decision is one-shot, and taking it does not change which decision you face next.

Which of five prompts, which price, which layout, which model for this class of request. If the action does not alter the state distribution there is no credit assignment, no discounting and no value propagation — the parts that make RL hard are all absent, and the method is thirty lines of code with a well-understood analysis. The arithmetic and an implementation are on multi-armed bandits.

The test: ask whether choosing option A now changes what situation arrives next. Serving a different button does not. Routing one request to a different model usually does not. Having an agent take a different first step in a ten-step task certainly does. If the answer is no, you are done, and you should be pleased about it.

3. Plain optimisation

When it applies: you can write down the objective and the constraints, and you have a model of the system.

Scheduling, routing, allocation, inventory, capacity planning. These look like sequential decision problems and are usually better handled by linear or mixed-integer programming, by dynamic programming, or by model predictive control — re-solving a short-horizon optimisation at every step using the current state.

The advantages over RL are not marginal. A solver returns the optimal solution for the model rather than an approximation of it. It reports infeasibility instead of silently returning something bad. Constraints are hard by construction rather than penalties an agent might decide to pay. And the result is inspectable: you can ask why, and get an answer, which matters enormously when a human has to sign off on the decision.

Its condition: a model of the system that is accurate enough. Where the dynamics are unknown or too complex to write down, RL becomes competitive — and the honest comparison is a solver against a fitted model versus RL against the real thing, not a solver against nothing.

4. Heuristics

When it applies: more often than anyone admits.

Retry twice, then escalate. Route anything over 8,000 tokens to the large model. Cache anything asked more than three times. These are policies. They are written in an afternoon, readable by everyone, and changed in a deploy rather than a training run.

A heuristic is also the baseline that makes every other number meaningful. If you cannot state what your rule-based policy achieves, you cannot say whether a learned one is an improvement, and a surprising number of learned policies have never been compared against the two-line rule they replaced. Build it first even if you intend to replace it.

The four conditions RL actually needs

All four, not any. If one is missing, one of the methods above is a better answer, and the missing condition tells you which.

The cases that satisfy all four are recognisable: games, robotic control in simulation with a transfer plan, reasoning training against verifiable rewards, and preference-based post-training where the reward model is the product of substantial separate work. The pattern across all of them is that somebody built the scoring infrastructure first, and the RL was the last step rather than the plan.

What RL costs that nobody budgets for

1. A reward function is a research project. It is not configuration. Getting it wrong produces a well-trained model of the wrong objective, and you find out at the end. This is where most of the calendar time goes and it is almost never in the estimate.
2. An environment is a software project, with tests, a maintainer, and bugs that are indistinguishable from agent failures until somebody looks. The environment page lists the five that cost the most time.
3. Sample requirements are large and scale badly. The arithmetic is on credit assignment: episodes needed grow with the inverse square of the effect size you need to resolve.
4. Results vary by seed. Henderson et al. (2018) documented performance differences between runs differing only in their random seed. A claim from one run is not a claim, which changes both how you evaluate and how long evaluating takes.
5. Debugging is genuinely hard. A flat learning curve is consistent with a broken environment, a wrong reward scale, a bad learning rate, insufficient exploration, or a task that is impossible from the given observation. Telling them apart is the content of most of this cluster.

None of that is an argument against reinforcement learning. It is the price of the one thing it does that nothing else does, and it is worth paying when you need that thing. The failure worth avoiding is paying it for a problem a contextual bandit, a solver or a rule would have handled in a week — which, on the evidence of how these projects usually start, is most of them.