RLHF with PPO

1

Concept

Reinforcement learning from human feedback (RLHF) treats the language model as a policy. A prompt defines an environment state, generated tokens are actions, and a completed response receives a reward predicted from human preferences. Unlike supervised fine-tuning, the policy is not asked to copy one reference answer. It explores its own responses and is updated toward outputs that score better.

A classic pipeline begins with a pretrained model, performs SFT, trains a reward model on preference pairs, and then optimizes a copy of the SFT policy. For each batch of prompts, the current policy samples responses. The reward model assigns scores. Additional terms may penalize undesirable properties or reward task checks. These returns are converted into advantages: estimates of how much better an action was than the policy’s expected baseline.

Language generation creates a credit-assignment problem. The reward often arrives after the full response, but hundreds of token choices contributed. A learned value function estimates expected future return at each position. Generalized advantage estimation can trade bias against variance when combining temporal-difference errors. The details matter because noisy advantages make large-model optimization unstable.

Proximal Policy Optimization (PPO) uses data sampled from an older policy to update a newer one. For a token action, let

Multiplying this ratio by the advantage encourages helpful changes. PPO’s clipped objective prevents the ratio from moving too far beyond a chosen interval in one update. If a positive-advantage token becomes vastly more likely, clipping stops that sample from demanding an unlimited step. The method is “proximal” because updates remain near the data-generating policy.

RLHF usually adds another anchor: a frozen reference policy, often the SFT checkpoint. A KL-divergence penalty discourages the optimized model from moving too far from the reference distribution. Without this pressure, the policy can sacrifice fluency, diversity, or general competence to exploit peculiarities of the reward model. A controller may adjust the penalty to target a desired KL range, but that target is an engineering choice, not a universal constant.

The training loop is expensive and stateful. It may hold policy, reference, reward, and value models, plus optimizer states and generated rollouts. Sampling is part of training, so throughput depends on inference infrastructure as well as backpropagation. Bugs in padding masks, terminal rewards, chat templates, old log-probabilities, or response boundaries can silently optimize the wrong objective.

Common symptoms include reward rising while independent quality falls, response length inflating, repetitive phrasing, excessive refusals, or KL collapsing or exploding. Evaluation must use held-out human judgments and task checks not reused as the optimized signal. Inspecting only the training reward is like grading an exam with the answer sheet the student learned to manipulate.

PPO remains important because it can optimize arbitrary sequence-level signals and incorporate online exploration. It is also sensitive to many hyperparameters and moving components. Direct preference methods remove the explicit reward-model-and-RL loop for some settings, trading flexibility for simplicity. Verifiable-reward methods keep reinforcement learning but replace or supplement a learned judge with programmatic checks.

The useful mental model is controlled policy search. The reward says which sampled directions look promising; the value model reduces variance; PPO clipping limits each update; and KL regularization keeps the search near a competent reference. None of these components removes the need to examine whether “higher reward” still means “better for the user.”