GRPO & RL on verifiable rewards

01 · Concept

Concept

Ask a reward model whether a patch fixes a bug and you get a learned judgment. Run the repository’s test suite against the patch and you get a reproducible pass/fail result under that suite. That difference — a learned estimate versus an executable observation with an explicit boundary — is the whole subject of this lesson, and it is why the strongest published claims about coding and mathematics now come from systems trained against checkers rather than judges. A passing suite is stronger evidence than an opinion, but it is not proof that the patch is correct beyond the behaviors the tests cover.

Verifiable rewards replace “does this sound good?” with a narrower question whose answer repeats. A math expression can be normalized and compared against a known result. Code can run against tests in a sandbox. A formal proof can be machine-checked. A game has rules and a score. Reinforcement learning fits this setting naturally, because the model can sample many candidate solutions and be told which worked without any human writing the desired reasoning path. A correct solution earns reward even when it takes a route no demonstration contained, which gives the policy room to discover strategies instead of merely imitating one.

Group Relative Policy Optimization (GRPO), from the 2024 DeepSeekMath work, samples a group of outputs for the same prompt. Let their rewards be $R_1,\ldots,R_G$. The group-relative advantage standardizes each sample against its peers:

Put numbers in it. Sample a group of eight attempts at one bug-fix task and run each through the test suite, giving rewards of 1 for two attempts and 0 for the other six. The mean is 0.25. The population standard deviation is $\sqrt{0.1875}\approx 0.433$. So each successful attempt gets an advantage of $(1-0.25)/0.433 \approx +1.73$, and each failure gets $(0-0.25)/0.433 \approx -0.58$. Two samples are pushed up hard, six are pushed down gently, and no learned value model was consulted anywhere — the group’s own mean was the baseline. Notice also that the advantages are automatically on a sane scale regardless of whether this task’s rewards were 0 and 1 or 0 and 100, which is what makes mixed-difficulty batches trainable.

Now the failure that follows from the same formula. Suppose the task is hard and all eight attempts fail: every reward is 0, the mean is 0, the standard deviation is 0, and every advantage is 0. The group cost you eight full generations and produced no gradient whatsoever. Flip it around — a task so easy all eight succeed — and you get the same nothing.

This is where the classic wrong turn happens. Seeing training plateau, a team pushes harder problems into the mix, reasoning that the model should be challenged. Throughput collapses, because a growing fraction of the compute budget is spent on all-fail groups that contribute nothing. The correct move is the opposite of intuition: you want problems where the policy succeeds sometimes, because contrast within the group is the entire learning signal. Curate a difficulty curriculum around the current success rate, track the fraction of groups with non-zero variance as a first-class training metric, and raise difficulty only as that fraction stays healthy. Group size, sampling temperature, and prompt selection are all levers on the same quantity.

The payoff for getting this right is the class of agentic coding results now being published. Qwen3.8-27B’s model card reports SWE-bench Pro 61.7, alongside Terminal Bench 2.1 at 73.0 and DeepSWE 1.1 at 42.2. Every one of those figures is vendor-reported and awaits independent reproduction — a caveat worth stating plainly rather than burying, because agentic benchmarks are unusually sensitive to harness configuration, retry policy, and scaffolding, and because no technical report accompanies this model to explain how the numbers were produced. Notice too that the spread between 61.7 and 42.2 across two coding benchmarks is far too large to treat any single score as “the model’s coding ability”. What the model card does establish is the shape of the claim: these are outcome scores from executable checkers, not preference scores from a judge, and that is a materially stronger kind of evidence than a win rate against another model’s prose.

Verifiability is not the same as a correct solution, though. An exact-answer checker accepts lucky guesses and reasoning built on invalid intermediate claims. Public unit tests can be hard-coded against. A code agent will exploit filesystem, timing, or network access if the sandbox permits it. A proof checker validates syntax and inference rules, never whether the theorem is the one the user wanted. Reward design has to state where the real boundary lies, and then defend it.

Rewards are usually composite: correctness, format validity, tool-use constraints, sometimes a learned style score. Their weights create incentives, and the incentives are rarely what you intended. A formatting reward can drown a sparse correctness signal. A length penalty can suppress the reasoning the task required. A learned judge folded into the mix reintroduces every hazard from lesson 6.4. Log each component separately, always, so that a rising total can never conceal a falling primary success rate.

Reward overfitting is the other durable risk. Narrow prompt templates and narrow checkers teach superficial strategies that evaporate on fresh distributions. Hold out problem generators rather than problems, mutate tests, include adversarial cases, and evaluate final answers and reasoning independently. Contamination between training problems and evaluation sets is especially corrosive here, because a memorized answer looks exactly like an improvement in reasoning.

GRPO is one member of a wider family and the acronym should not obscure the principle: sample a group, estimate which members did relatively well, update conservatively. It is compelling wherever a high-precision checker can supply feedback at scale, and unsuitable wherever the desired outcome is plural, value-laden, or resistant to a robust programmatic test. The durable model is a tournament with an audited referee — the group creates local competition, the verifier scores it, optimization increases the probability of winners — and the learning is only as good as the referee’s grasp of what users actually care about.