Curriculum Learning and Task Ordering

A curriculum orders training tasks so the model meets easy ones before hard ones. In supervised learning the evidence for this is mixed. In reinforcement learning with sparse rewards it is often the difference between learning and not learning at all, and the reason is arithmetic rather than pedagogy.

The claim, and where it came from

The idea was formalised for machine learning by Bengio et al. (2009), “Curriculum Learning”, which framed it as a continuation method: start with a smoothed, easier version of the objective and gradually restore the real one, so that early optimisation lands in a better basin.

For supervised learning, results since have been genuinely mixed. Random shuffling is a strong baseline, large models are relatively insensitive to ordering, and reported gains are often small and dataset-specific. Treat “order your fine-tuning data by difficulty” as a hypothesis to test on your own data rather than as established practice.

Reinforcement learning is a different situation, because there the ordering does not merely affect the optimisation path. It affects whether there is any gradient at all.

Why it matters much more in RL

A supervised example always produces a gradient. Every image has a label; every token has a target; a hard example produces a large loss and a large gradient, which is the opposite of nothing.

A reinforcement learning episode on a task the agent never solves produces a reward of zero, an advantage of zero after baselining, and a gradient of zero. The agent does not learn slowly from impossible tasks. It learns nothing from them, and it burns the same compute doing it.

That is the whole argument, and it explains why sparse-reward environments and curriculum methods appear together so consistently in the literature. The curriculum is not making learning more efficient. It is making the reward reachable often enough that learning can start.

Which difficulty is most informative

For a binary outcome — solved or not — with success probability p, the variance of the reward is p(1 - p). That function peaks at p = 0.5, where it equals 0.25, and falls to zero at both ends.

p        p(1-p)     relative information
0.05      0.0475          19%
0.20      0.1600          64%
0.50      0.2500         100%
0.80      0.1600          64%
0.95      0.0475          19%

Variance in the reward is the signal. A task the agent always fails and a task it always solves both produce a constant, and a constant carries no information about which action was better. The most informative tasks are the ones the agent gets right about half the time — a quantitative statement of something teachers have always asserted, with the useful property that you can measure it.

Under group-based methods the effect is sharper still, because a whole group of samples must contain disagreement. The GRPO page works the numbers: at a group size of 8, a task with a 90 per cent pass rate wastes 43 per cent of its groups on unanimous outcomes, and at 99 per cent it wastes 92 per cent. Curriculum design in that setting is not a refinement, it is most of the compute budget.

Automatic curricula

Hand-ordering tasks requires knowing their difficulty for the agent, which is not the same as their difficulty for you. The methods that work best measure it instead.

Two different things called curriculum

In language model training the word covers two practices that behave quite differently, and conflating them is why the evidence looks contradictory.

• Difficulty ordering — sorting examples from easy to hard within a fixed dataset. This is the original sense, and it is the one whose supervised results are mixed. Large models see each example few times, gradients are averaged over large batches, and the ordering effect that mattered for small models trained for many epochs largely washes out. Test it if you like; do not assume it.
• Mixture scheduling — changing the proportions of data sources over training, such as raising the share of high-quality or domain-specific data towards the end of a run. This is standard practice and it is a different mechanism: it is not about the order in which a fixed set is seen but about which distribution the final gradients come from. The last data a model sees has disproportionate influence on its final weights, which is a fact about optimisation rather than about pedagogy.

For RL post-training, neither of those is the operative version. What matters there is difficulty selection against the current policy — keeping the prompt pool in the band where the model sometimes succeeds — and that is not an ordering at all. It is a filter that must be re-applied as the policy improves, because the band moves underneath it.

The practical distinction to hold on to: ordering a fixed dataset is a hypothesis, adjusting the mixture over a long run is established, and selecting RL prompts by measured pass rate is close to mandatory because without it a large share of the compute produces no gradient.

Three ways it backfires

1. The agent overfits the easy distribution

Train long enough on easy tasks and the policy specialises in them, including in ways that do not survive contact with hard ones. A robot taught on flat ground learns a gait that only works on flat ground; a model trained on short problems learns strategies that do not extend. The curriculum was meant to be a scaffold and became the training set.

2. Earlier skills are forgotten

Moving on from a stage does not preserve the ability acquired in it. This is catastrophic forgetting in a curriculum-shaped disguise, and the standard fix is the standard one: keep sampling from earlier stages at some low rate rather than retiring them. A mixture is safer than a schedule.

3. The difficulty ordering is wrong

A hand-built curriculum encodes the designer’s theory of what is hard, and that theory is often wrong about a model. Problems people find easy can be hard for a policy and the reverse happens constantly. Every measured curriculum — pass-rate filtering, self-play — avoids this failure entirely, which is the strongest argument for preferring them.

What to do about it in practice

1. Measure the pass rate of your current policy across the task pool before writing any curriculum logic. If most of the pool is already at 0 or 1, you have found your problem and no hyperparameter will fix it.
2. Train on the middle band and re-measure every few thousand steps. This is the whole method for most practical purposes and it needs about fifty lines of code.
3. Keep a fixed evaluation set drawn from the target distribution, never from the curriculum. Otherwise progress on the curriculum is indistinguishable from progress on the task.
4. Keep a low rate of earlier and harder tasks in the mixture. Earlier tasks resist forgetting; harder ones tell you when the band should move.
5. Log the distribution of rewards, not just the mean. A mean of 0.5 from tasks all at 0.5 and a mean of 0.5 from half at 0 and half at 1 are completely different training situations, and only one of them is producing gradient.

The last point generalises past curricula. In any RL run, the reward histogram is more diagnostic than the reward mean, and it is the first thing to plot when a run is quietly doing nothing.