GRPO & RL on verifiable rewards

1

Concept

Preference models are useful when quality is subjective, but some tasks have outcomes a program can check. A math expression can be normalized and compared with a known answer. Code can run against tests in a sandbox. A formal proof can be checked by a verifier. A game has rules and a score. These verifiable rewards replace “does this sound good?” with a narrower question whose answer is repeatable.

Reinforcement learning is attractive here because the model can sample many candidate solutions and receive feedback on outcomes without a human writing the exact desired reasoning trace. A correct response earns reward even if its path differs from a reference demonstration. This gives the policy room to discover strategies rather than merely imitate one solution.

Group Relative Policy Optimization (GRPO), introduced in the 2024 DeepSeekMath work, samples a group of outputs for the same prompt. Let their rewards be $R_1,\ldots,R_G$. A simplified group-relative advantage standardizes a sample against the group:

Outputs above the group baseline receive positive advantage; those below receive negative advantage. The exact implementation can include token-level policy ratios, clipping, KL regularization, and multiple reward components. The key distinction from classic PPO pipelines is that the group statistics provide a baseline, so a separate learned value model is not required in the basic design.

Relative comparison helps when raw reward scales differ across prompts. Solving one unusually hard problem can still be reinforced against unsuccessful peers, even if rewards are sparse. But groups consume generation compute. If every sample receives the same reward, the normalized learning signal vanishes. Group size, sampling diversity, and prompt difficulty curriculum influence how often useful contrast appears.

Verifiability is not the same as correctness of the entire solution. An exact final-answer checker can accept lucky guesses or reasoning with invalid intermediate claims. Public unit tests can be hard-coded. A code agent may exploit filesystem, timing, or network access if the sandbox is weak. A proof checker validates formal syntax and rules, not whether the theorem represents the user’s intent. Reward design must define the real boundary.

Multiple rewards are often combined: correctness, format validity, tool constraints, and perhaps a learned style score. Their weights create incentives. A formatting reward can overwhelm sparse correctness; a length penalty can suppress necessary reasoning; a judge can reintroduce reward hacking. Log each component separately so a rising total cannot hide a falling primary success rate.

Training also risks reward overfitting. If prompts and checkers have narrow templates, the policy can learn superficial strategies that fail on fresh distributions. Hold out problem generators, mutate tests, include adversarial cases, and evaluate final answers and reasoning independently. Contamination between training problems and evaluation sets is especially damaging because repeated answers can mimic reasoning improvement.

GRPO is one member of a wider family, and the acronym should not obscure the principle: compare sampled behavior, estimate which actions performed relatively well, and update conservatively. It is particularly compelling when a high-precision checker supplies scalable feedback. It is less suitable when the desired outcome is inherently plural, value-laden, or impossible to reduce to a robust programmatic test.

The durable mental model is a tournament with an audited referee. The group creates local competition; the verifier supplies scores; policy optimization increases the probability of winners. Better learning follows only when the referee checks what users actually care about and the tournament covers the situations the model will face.