Reinforcement Learning in One Page

Reinforcement learning is what you do when you can score an outcome but cannot demonstrate the behaviour that produces it. Everything else in the field — value functions, policy gradients, PPO — is machinery for turning that score into a gradient.

The definition, and the four objects

A reinforcement learning problem has four parts, and you can check any proposal against them in a minute.

Two derived terms carry most of the remaining weight. An episode is one run from a starting state to a terminal one. The return is the sum of rewards over an episode, usually discounted so that a reward now counts for more than the same reward later. The agent is optimising expected return, not reward — the difference is the whole reason value functions exist.

What makes this different from supervised learning is not the presence of feedback. It is that the agent’s own actions determine what data it sees next. Change the policy and you change the distribution of states it visits, which changes the gradient, which changes the policy. Supervised learning has a fixed dataset. Reinforcement learning has a dataset that argues back.

The same four objects in LLM terms

If you have shipped anything on top of a language model, you have built three of these already under different names.

• The policy is the model. A language model maps a token sequence to a distribution over the next token. That is exactly the type signature of a policy over a discrete action space of vocabulary size, and it is why RL post-training needs no architectural change at all — the network was already a policy.
• An action is a token — or, at a coarser grain, a whole tool call. Which grain you choose decides how hard credit assignment gets.
• An episode is a rollout. Generating one completion from one prompt is one episode. Sampling eight completions for the same prompt is eight episodes sharing a start state, which is the observation GRPO is built on.
• The reward is a learned scoring function. A reward model is a network with the language modelling head swapped for a single scalar output. It is trained on preference comparisons and it plays the part the game score plays in Atari.

The environment is the one piece that has no obvious counterpart, and that is the honest difficulty. In chat post-training the environment is trivial — a prompt goes in, a completion comes out, the episode ends — which is why RLHF looks so unlike textbook RL. In agentic training the environment is a real thing with state, and building it is most of the work. That is what writing your own environment is about.

The loop, in eight lines

Every reinforcement learning system, from a bandit to a robot, is this loop with different things plugged into it.

obs, info = env.reset()
done = False
while not done:
    action = policy(obs)                       # the agent decides
    obs, reward, terminated, truncated, info = env.step(action)
    learner.observe(obs, action, reward)       # the agent records
    done = terminated or truncated
learner.update()                               # the agent improves

The five-value return from step is worth reading closely. terminated means the episode ended for a reason internal to the problem: the agent reached the goal, or died, or the conversation finished. truncated means it ended for a reason external to the problem: a step limit, a wall clock, a budget. They are separate because the value of the final state differs. A terminated episode has no future to bootstrap from; a truncated one does, and treating a time-out as a terminal state teaches the agent that running out of time is a natural end to the task.

A gridworld that runs

Below is a complete reinforcement learning system: a four-by-four grid with two pits and a goal, tabular Q-learning, and a printout of the learned policy. No dependencies beyond the standard library. It is under forty lines because the ideas are small; everything after this in the field is about making the same ideas work when the table will not fit in memory.

import random

GRID = ["S...",
        ".X.X",
        "....",
        "X..G"]
N = len(GRID)
MOVES = [(-1, 0), (0, 1), (1, 0), (0, -1)]   # up, right, down, left
ARROWS = "^>v<"

def step(state, action):
    r, c = state
    dr, dc = MOVES[action]
    nr, nc = r + dr, c + dc
    if not (0 <= nr < N and 0 <= nc < N):
        nr, nc = r, c                        # walls bounce you back
    cell = GRID[nr][nc]
    if cell == "G":
        return (nr, nc), 1.0, True
    if cell == "X":
        return (nr, nc), -1.0, True
    return (nr, nc), -0.02, False            # a small cost per move

def train(episodes=5000, alpha=0.5, gamma=0.95, eps=0.2, seed=0):
    rng = random.Random(seed)
    Q = {(r, c): [0.0] * 4 for r in range(N) for c in range(N)}
    for _ in range(episodes):
        state, done = (0, 0), False
        for _ in range(100):                 # truncate a wandering episode
            if done:
                break
            if rng.random() < eps:
                a = rng.randrange(4)         # explore
            else:
                a = max(range(4), key=lambda i: Q[state][i])   # exploit
            nxt, reward, done = step(state, a)
            target = reward if done else reward + gamma * max(Q[nxt])
            Q[state][a] += alpha * (target - Q[state][a])
            state = nxt
    return Q

Q = train()
for r in range(N):
    print("".join(GRID[r][c] if GRID[r][c] in "XG"
                  else ARROWS[max(range(4), key=lambda i: Q[(r, c)][i])]
                  for c in range(N)))

It prints a grid of arrows: a route from the top-left corner to the goal that goes around the pits. Where two routes are the same length the arrows depend on the seed, because the two policies are genuinely equally good and nothing in the reward distinguishes them. That is worth noticing early — an RL system optimises the reward you wrote, and is indifferent to everything you did not write down.

Three parameters do all the work here. alpha is how much of each new estimate to keep, gamma is how much a later reward is worth relative to an earlier one, and eps is how often the agent tries something other than its current best guess. Change the step cost from -0.02 to 0 and the agent stops caring about the length of the route.

Why value is a separate idea from reward

Reward is what you get now. Value is what you expect to get from here on, following your current policy. They are different numbers and confusing them is the most common early mistake.

The square immediately above the goal has a reward of -0.02 — it is not a good place in itself. Its value is high, because from there one action reaches a reward of 1. Learning is almost entirely the process of value flowing backwards from the places where reward actually appears, one step per update, which is a thing you can watch happen in the Q-table on the Q-learning page.

This is also why a sparse reward is hard rather than merely slow. If the reward is zero everywhere except at a goal 400 steps away, the value signal has to propagate 400 steps backwards through updates that are each mostly noise, and the agent has to stumble into the goal at least once by accident before there is anything to propagate.

What is not reinforcement learning

The term has spread to cover a great deal that is not it, and the distinctions are load-bearing when you are deciding what to build.

• Learning from feedback is not automatically RL. Collecting thumbs-up ratings and fine-tuning on the good ones is supervised fine-tuning with a filter. There is no policy-dependent data distribution and no credit assignment, and that is a feature: it is far cheaper.
• DPO is not reinforcement learning in the mechanical sense, despite the lineage. It optimises a supervised loss on a fixed set of preference pairs with no sampling in the loop, which is exactly what makes it cheap. The comparison lives on DPO versus PPO.
• A model calling tools in a loop is not an RL agent. The word agent means two different things in the two fields. In an LLM agent loop nothing is being trained; the policy is frozen and the loop is inference.
• One-step decisions are bandits, not RL. If your action does not change what situation you face next, you have a bandit problem, which is a solved and much easier one.

The reason to be strict about this is cost. Reinforcement learning buys you the ability to optimise a behaviour you cannot demonstrate, and it charges for it in sample efficiency, reward engineering and reproducibility. Four cheaper methods beat it on most real problems, and it is worth knowing which one applies before writing a reward function.