Offline RL and Learning From Logged Data

Offline reinforcement learning trains a policy from a fixed dataset of past interactions, with no environment to try things in. It is the version of RL that fits how most companies actually hold data, and it fails in a specific way that has nothing to do with having too little of it.

The promise, and why it is so attractive

You have logs. Every request that came in, what your system did, and how it turned out. Offline RL says: treat those as trajectories, learn a better policy from them, deploy it. No simulator to build, no risk of an untrained agent touching production, and the data already exists.

The theory is encouraging too. Q-learning is off-policy — its update assumes optimal continuation regardless of what the behaviour policy actually did — so in principle it can learn the value of the best policy from data generated by a mediocre one. Levine et al. (2020) wrote the standard tutorial and review on this setting and it remains the best single entry point.

In practice, running standard off-policy algorithms on a fixed dataset usually produces a policy that scores brilliantly on its own value estimates and badly in reality. The reason is precise.

The trap: extrapolation plus a maximum

The Q-learning target contains max over all actions in the next state. With a table, every one of those values was learned from data. With a network and a fixed dataset, most of them were not: the behaviour policy never took those actions, so the network is extrapolating.

1. The network produces a value for an action never taken in that state. It is a guess from a function approximator, so it is wrong in some direction with some magnitude.
2. The max operator selects across actions. Taking the maximum of a set of noisy estimates does not select the best action — it selects the largest error. Overestimates are systematically preferred.
3. That inflated value becomes the regression target for the previous state. The error is now in the data.
4. Repeat. Each iteration propagates the inflation backwards and adds a new one. Values diverge upwards, sometimes to absurd magnitudes, while the loss looks fine because the network is fitting its own targets.

Online RL is protected from this by the environment. Take the overestimated action, get a disappointing reward, correct the estimate. The correction loop is the environment, and offline RL removes it. This is the single mechanism behind most offline RL failures, and it is why more data of the same kind does not help: the problem is the actions the data does not contain.

Why off-policy evaluation collapses

A related question is whether you can at least evaluate a new policy on old data without deploying it. The textbook answer is importance sampling: weight each logged trajectory by how much more likely the new policy was to have produced it.

weight(trajectory) = product over steps of  pi_new(a_t | s_t) / pi_log(a_t | s_t)

That product is the problem, and the arithmetic is brutal:

suppose the new policy is twice as likely to take each logged action:

  1 step   ->  weight = 2
  5 steps  ->  weight = 2^5   =      32
 10 steps  ->  weight = 2^10  =   1,024
 20 steps  ->  weight = 2^20  = 1,048,576

and if the new policy is HALF as likely at each step:

 20 steps  ->  weight = 2^-20 = 0.00000095

With weights spanning six orders of magnitude, the estimate is dominated by a handful of trajectories. A useful way to see how bad it has become is the effective sample size, roughly (sum of weights)^2 / (sum of squared weights). A dataset of 100,000 logged episodes can have an effective sample size in the single digits once the weights spread out, which means your evaluation of the new policy rests on about five episodes regardless of how many you stored.

The mitigations — clipping the weights, per-step rather than per-trajectory weighting, doubly robust estimators combining a model with importance weights — all reduce variance by adding bias. None rescues an evaluation where the new policy is genuinely different from the logged one. For short horizons, and especially for one-step decisions, off-policy evaluation works well, which is another argument for keeping a problem in bandit form when you can.

The three families of fix

The common thread is that all three trade attainable performance for reliability. This is the correct trade in an offline setting and it is worth stating plainly to anyone expecting offline RL to discover a strategy nobody has tried: it will not, and the methods that work are the ones that stop trying.

Two things wrong with your logs specifically

Before any of the above applies, production logs have two properties that make them worse than a benchmark dataset.

They contain no exploration. Your system did what it was configured to do. If it always routed long requests to the large model, there is not one example of a long request going to the small one, so no method can estimate what would have happened. The fix is cheap and has to be decided in advance: send a small fraction of traffic — one or two per cent — to a randomised choice, and log the propensity you used. A log with recorded propensities is a dataset; a log without them is a transcript.

They are confounded by the decisions that produced them. Suppose you route hard requests to the expensive model and easy ones to the cheap model. In the logs, the expensive model has a lower success rate, because it only ever saw the hard cases. A naive learner concludes the expensive model is worse and routes everything away from it. The confound is not in the data, it is in the policy that generated the data, and no amount of it will reveal the mistake.

What this means for LLM systems

The language model world arrived at the same conclusion from a different direction and mostly stopped bootstrapping. Direct preference optimisation is offline learning that avoids every problem on this page by not learning a value function at all: a fixed set of preference pairs, a supervised loss, no maximum over unseen actions and nothing to diverge. The trade — that it cannot exceed what the preference data contains — is precisely the trade offline RL forces anyway, made explicit. That comparison is on DPO versus PPO.

For an agent system with logged trajectories, the practical hierarchy is worth stating in order. Filtering the logs to successful episodes and fine-tuning on them is behaviour cloning, and it works. Ranking logged outcomes into preference pairs and running a preference method is a step up and still avoids bootstrapping. Full offline RL over multi-step trajectories is the last resort, and it is worth attempting only when you can point at a decision whose value genuinely depends on what happens several steps later — which is rarer than it sounds, and is the argument of the last page in this cluster.