DPO: skipping the reward model

1

Concept

PPO-based RLHF has a long chain: collect preferences, fit a reward model, sample from a policy, estimate advantages, update a value model, constrain the policy, and repeat. Direct Preference Optimization (DPO) starts from a mathematical relationship between a KL-regularized optimal policy and its reward. It rearranges that relationship so the policy itself can be trained directly on preference pairs, without fitting an explicit reward model or running a conventional RL loop.

Each example still contains a prompt $x$, preferred response $y^+$, and rejected response $y^-$. DPO uses a trainable policy $\pi_\theta$ and frozen reference $\pi_{ref}$. For each response, it sums token log-probabilities conditioned on the prompt. The relevant margin compares how much more the trainable policy favors each response than the reference does:

The loss $-\log\sigma(m)$ increases the chosen response’s relative advantage and decreases the rejected response’s. The coefficient $\beta$ controls the strength of the reference-relative preference scale in the standard formulation. Implementation conventions can invert or reparameterize it, so copying a numeric setting across libraries without checking definitions is unsafe.

The word relative is crucial. DPO does not merely maximize the chosen response’s raw likelihood. It asks the policy to favor it more than the reference policy already did, compared with the rejected response. The reference supplies an anchor analogous to KL regularization in the derivation. If policy and reference are accidentally the same live object, or if log-probabilities are computed under different chat templates, the intended objective breaks.

DPO is operationally attractive. Training resembles supervised learning over fixed sequences: perform forward passes, gather response-token log-probabilities, compute a pairwise loss, and backpropagate. There are no on-policy rollout workers, no learned value head, and no explicit reward-model checkpoint. This can reduce engineering complexity and variance.

But “skipping the reward model” does not skip preferences. All weaknesses of the comparison data remain: annotator disagreement, length bias, limited prompt coverage, stale candidate policies, and mislabeled examples. DPO can overfit stylistic shortcuts. It can also reduce likelihood for rejected responses that contain useful portions, because the label applies to the full sequence rather than identifying the exact failing span.

Offline data creates a further boundary. Standard DPO does not sample the changing policy and ask for new judgments during training. If the policy moves into unseen behavior, the fixed dataset offers no direct feedback there. Iterative variants can regenerate candidates and collect new pairs, but that restores a data-production loop even if it still avoids PPO.

Sequence length and reduction choices deserve attention. Summed log-probabilities naturally scale with the number of response tokens; alternative normalization can change incentives. Padding must be masked, prompt tokens excluded consistently, and chosen/rejected truncation must not delete the evidence that determined the label. Logging margins, accuracies, lengths, and reference divergences helps reveal silent failures.

The durable mental model is pairwise policy shaping. DPO makes the desired response more characteristic of the new policy relative to a fixed baseline, and makes the rejected response less so. It replaces a learned scalar judge and online optimizer with an elegant offline loss—but the human judgments, reference choice, and evaluation remain the real specification.