RLHF with PPO

01 · Concept

Concept

Supervised fine-tuning can only teach the model to reproduce answers somebody already wrote. If you want it to find a better answer than anything in your dataset, it has to generate candidates itself and be told which ones worked. That is a reinforcement-learning problem, and reinforcement learning from human feedback (RLHF) is the classical way of posing it. The prompt is a state, generated tokens are actions, and a completed response earns a reward predicted from human preferences.

A canonical pipeline runs pretraining, then SFT, then a reward model trained on preference pairs, then optimization of a copy of the SFT policy. For each batch of prompts the current policy samples responses, the reward model scores them, and those returns are converted into advantages: estimates of how much better an action was than the policy’s own expected baseline.

Language generation makes credit assignment hard. The reward usually arrives once, after the full response, yet hundreds of token choices contributed to it. A learned value function estimates expected future return at each position, and generalized advantage estimation trades bias against variance when combining temporal-difference errors. These details are not decoration: noisy advantages make large-model optimization unstable in ways that look like model problems.

Proximal Policy Optimization (PPO) lets you reuse data sampled from a slightly older policy. For a token action,

Work the clip through with numbers, because its asymmetry surprises people. Take a clip range of $\epsilon=0.2$, so the clipped copy of the ratio lies between 0.8 and 1.2. Suppose one token had probability 0.20 under the old policy and 0.30 under the new one, giving $\rho_t = 1.5$, and its advantage was $+2.0$. The unclipped term is 3.0, the clipped term is 1.2 times 2.0, or 2.4, and the minimum of the two is 2.4. The surrogate objective gains nothing further by pushing this token’s probability higher, so that sample’s incentive to keep climbing disappears. The ratio itself is still 1.5: PPO did not clamp it to 1.2. Now keep the same ratio but make the advantage $-2.0$. The unclipped term is $-3.0$, the clipped term is $-2.4$, and the minimum is $-3.0$. The clip does not flatten the objective in this direction — a bad action whose probability has risen keeps being pushed down hard. Other samples share the same parameters, so their gradients can move this ratio farther outside the nominal range. PPO clips an objective contribution, not the optimizer step or the resulting policy.

RLHF adds a second anchor beyond clipping: a frozen reference policy, usually the SFT checkpoint, with a KL-divergence penalty discouraging the optimized model from drifting away from it. Clipping reshapes the local incentive; KL penalizes accumulated drift. Neither is a hard bound. Without the KL term, the policy can sacrifice fluency, diversity, and general competence to exploit peculiarities of the reward model. A controller may adjust the penalty coefficient to target a KL band, but that band is an engineering choice, not a constant of nature.

Here is the classic wrong turn, and almost everyone takes it once. The dashboard shows mean reward climbing from 0.9 to 1.3 over a training run, roughly a 40 percent improvement, and the run is declared a success. Lesson 6.4 already dismantled this: the reward model’s pairwise loss depends only on score differences, so the scale has no unit and the level has no meaning. Worse, the quantity going up is measured by exactly the model the policy is learning to game. The correction is to treat training reward as a diagnostic and never as an outcome. Judge the run on held-out human comparisons collected after training, on task checks that were never part of the optimized signal, and on the control variables — KL against the reference, mean response length, refusal rate, sample diversity. A run where reward rose 40 percent while length rose 60 percent and KL doubled did not get better; it got longer and stranger.

The loop is also expensive in a way that quietly excludes most teams. A full PPO setup can hold four models at once — policy, frozen reference, reward, and value — plus optimizer state and generated rollouts. Sizing that for a 27B-class model is sobering: the policy alone is 54 GB in bf16, the reference is another 54 GB, and AdamW state for 27B parameters runs somewhere between 324 and 432 GB. Sampling is part of training, so throughput depends on your inference stack as much as on backpropagation. And because so many components move together, bugs hide well: mismatched padding masks, terminal rewards attached to the wrong position, stale old log-probabilities, a chat template applied during sampling but not during scoring. Any of these silently optimizes a different objective than the one you wrote down.

The visible symptoms are consistent across teams. Reward rises while independent quality falls. Response length inflates. Phrasing becomes repetitive. Refusals multiply. KL collapses toward zero, meaning nothing is being learned, or explodes, meaning the policy has left the language distribution. Evaluate with judgments and checks that were never part of the optimized signal, or you are grading an exam with the answer sheet the student learned to forge.

PPO remains important because it can optimize arbitrary sequence-level signals and explore online, which no offline method can. It is also sensitive to many hyperparameters across many moving parts. Direct preference methods (lesson 6.6) delete the reward model and the RL loop for a class of problems, trading flexibility for simplicity. Verifiable-reward methods (lesson 6.8) keep the reinforcement learning but replace the learned judge with a programmatic checker, which changes the risk profile entirely. Whichever you pick, the mental model holds: the reward proposes a direction, the value model reduces variance, clipping limits the incentive to keep moving in selected directions, and KL penalizes drift — none of it hard-bounds the policy, and none answers whether “higher reward” still means “better for the person reading the output”.