Skip to content
Multigrid

Reinforcement learning

The field behind RLHF, explained for an engineer who knows language models and has never trained an agent: policies, rewards, value, and the arithmetic underneath each.

Most engineers now meet reinforcement learning backwards. They learn that a chat model was tuned with RLHF, that RLHF uses PPO, and that PPO has a clipped objective — three names deep into a field whose first principles they never saw. The result is a working vocabulary sitting on no foundation, which holds up until something goes wrong.

These pages build the foundation from the other end. An environment, an agent, a reward, and a loop between them; then value, then the two families of algorithm that estimate it; then the specific machinery that language model post-training uses and why it looks the way it does. The translation is made explicit throughout, because it is short: a policy is a model, an episode is a rollout, a reward model is a learned scoring function, an action is a token.

Where the standard explanation reaches for notation, these pages reach for a worked number instead. A Q-table is updated by hand until the values move. A bandit allocation is computed over three batches. The last page in the cluster argues that most problems people bring to reinforcement learning should be solved with something cheaper, and says which something.

Reinforcement Learning in One Page

Agent, environment, reward and policy defined by what each one does in the loop, with a complete gridworld in forty lines of dependency-free Python.

9 min read

Markov Decision Processes Without the Notation Wall

States, actions, transitions, rewards and discounting worked through one concrete example — routing a query across two models under a budget.

9 min read

Q-Learning, Step by Step

A four-state Q-table updated by hand through five episodes, every number shown, then compared against the exact optimal values.

10 min read

Policy Gradients and REINFORCE

How you take a gradient through a sampling step at all, the log-derivative trick that makes it possible, and the REINFORCE update in twelve lines of PyTorch.

10 min read

Actor-Critic Methods: Two Networks, One Loop

What the critic predicts, how the advantage is computed on a real three-step episode, and what the lambda in GAE does to the numbers.

9 min read

PPO: The Algorithm Behind Most RLHF

The clipped objective explained through the failure it prevents, with the four cases of the min worked out, plus what PPO looks like in a language model pipeline.

11 min read

Reward Models: Turning Preferences Into a Number

How pairwise comparisons become a scalar score, why the absolute value of that score means nothing, and the two places the scale drifts without warning.

10 min read

Reward Hacking, With Documented Examples

Six cases from named, dated published work, each explained by the mechanism that produced it rather than told as an anecdote.

10 min read

Exploration and Exploitation, Made Concrete

The trade-off stated as regret, four strategies separated by their arithmetic, and why exploration is a much harder problem in an MDP than in a bandit.

9 min read

Multi-Armed Bandits for Product Decisions

An A/B test that reallocates traffic as it learns, with the epsilon-greedy allocation worked over three batches and a thirty-line Thompson sampling implementation.

10 min read

Offline RL and Learning From Logged Data

Why a Q-function trained on logs overestimates actions nobody took, how the error compounds through bootstrapping, and what the standard fixes actually constrain.

10 min read

RLAIF: The Model as Preference Labeller

How AI feedback replaces human preference labels, what the published comparisons actually claim, and the biases a single labeller model concentrates rather than averages out.

10 min read

Group-Relative Policy Optimisation

What GRPO removes from PPO, how a group of samples replaces the value network, and the arithmetic that shows when a group produces no learning signal at all.

10 min read

Verifiable Rewards: Training on Problems With Right Answers

Why reasoning training started with maths and code, what an exact checker changes about reward hacking, and the ways a verifier can still be gamed.

10 min read

Curriculum Learning and Task Ordering

Why ordering training tasks by difficulty helps when reward is sparse, the arithmetic that says which difficulty is most informative, and the three ways a curriculum backfires.

9 min read

Self-Play, and Where It Applies Outside Games

The three properties that make self-play work in games, why most real tasks have only one of them, and what an honest transfer to non-game problems looks like.

9 min read

Writing Your Own RL Environment

A complete environment class with reset and step, a random-agent loop, a tabular learner that beats it, and the five mistakes that make an environment untrainable.

12 min read

Credit Assignment Over Long Horizons

Why one scalar at the end of 400 steps cannot say which step earned it, worked as a signal-to-noise calculation, and the five techniques that attack it.

9 min read

Safe RL and Constrained Optimisation

Penalty terms versus hard constraints, the arithmetic showing when a penalty stops binding, and the distinct failure modes of each approach.

9 min read

Four Cheaper Methods That Beat RL on Most Real Problems

Supervised learning on logged decisions, bandits, plain optimisation and heuristics — what each one needs, and the four conditions that must all hold before RL is the right answer.

10 min read

Other topics