DPO: skipping the reward model

01 · Concept

Concept

Lesson 6.5 left you holding four models at once and a 27B-parameter policy whose optimizer state alone runs to hundreds of gigabytes. Most teams cannot run that, and many do not need to. Direct Preference Optimization (DPO) starts from a mathematical relationship between a KL-regularized optimal policy and its reward, then rearranges it so the policy can be trained directly on preference pairs — no reward model, no rollout workers, no value head. What survives is a forward pass and a loss.

Each example still contains a prompt $x$, a preferred response $y^{+}$, and a rejected response $y^{-}$. DPO uses a trainable policy $\pi_\theta$ and a frozen reference $\pi_{ref}$, sums token log-probabilities over each response conditioned on the prompt, and compares how much more the policy favours each response than the reference already did:

Now make it concrete on this course’s model. You want to adapt Qwen3.8-27B’s instruct-mode style for a support product, and you have 8,000 preference pairs. Setting up the run means answering four questions, in this order.

First, where does the reference model come from? Naively loading two 27B bf16 checkpoints is not a one-card plan. Combine DPO with LoRA from lesson 6.3 instead: train adapters over one low-bit backbone and obtain the reference log-probability by running the same backbone with adapters disabled. That removes the second weight copy, but it does not prove a fixed GPU fit; activation memory, quantization metadata, sequence length, micro-batch, and backend buffers still determine residency.

Second, which mode are you training? Qwen3.8-27B ships thinking mode on by default and instruct mode behind enable_thinking set to False, and the two render the same conversation into different token sequences. Log-probabilities are computed over tokens, so a pair collected in one mode and scored in the other is not merely noisy — it is a different objective. Pick a mode, render both the chosen and rejected responses through that exact template branch, and record which branch the dataset used.

Third, which tokens enter the sum? If you are training thinking mode and your chosen response carries a 900-token reasoning trace while the rejected one carries 200, the summed log-probability difference is dominated by trace length before quality gets a vote. Either mask the thinking segment out of the loss and score only the final answer, or length-control your pairs deliberately. Do not leave it to chance.

Fourth, how were the candidates sampled? Record the full configuration. To reproduce the model card’s recommendations, thinking uses temperature 1.0, top_p 0.95, top_k 20, min_p 0.0, presence_penalty 0.0, and repetition_penalty 1.0; instruct uses 0.7, 0.80, 20, 0.0, 1.5, and 1.0. The deployment operator can choose other settings, so off-recommendation samples are not intrinsically invalid or responses users will never see. What matters is whether the candidate distribution covers the behavior you intend to deploy and whether that choice is recorded.

Then run one example through the arithmetic. Take $\beta = 0.1$. Suppose the chosen response scores $-42.0$ under the policy and $-45.0$ under the reference, a relative gain of $+3.0$; the rejected response scores $-50.0$ under the policy and $-48.0$ under the reference, a relative loss of $-2.0$. Then $m = 0.1 \times (3.0 - (-2.0)) = 0.5$, and the loss is $-\log\sigma(0.5)\approx 0.47$. Note what moved the number: not how likely either response is, but how each one’s likelihood shifted relative to the reference.

That distinction sets up the classic wrong turn. Teams log the policy’s raw log-probability of the chosen responses, watch it fall during training, and conclude the run is broken — the model is getting worse at producing the answers we told it to prefer. Look at both series and the panic dissolves. It is entirely normal for chosen log-probability to drift down from $-42$ to $-44$ while rejected log-probability collapses from $-50$ to $-60$: the margin widened from 8 to 16, which is exactly what the objective optimizes. DPO never promised to raise the chosen response’s absolute likelihood. It promised to make the chosen response more characteristic of the new policy than the rejected one is. If you want an absolute likelihood floor, you need a supervised term as well — which is precisely the design ORPO adopts, in lesson 6.7. What you should watch instead is the margin, the pairwise accuracy, the mean response length, and the KL-like divergence from the reference; when chosen log-probability falls off a cliff and length collapses with it, the run really is degenerating.

Removing the reward model does not remove the preferences, and every weakness lesson 6.4 catalogued survives intact: annotator disagreement, length bias, thin prompt coverage, stale candidate policies, mislabelled examples. DPO can also punish a rejected response that contained useful material, because the label applies to the whole sequence and never identifies the failing span. And because standard DPO is offline, it collects no new judgments as the policy moves; if training pushes the model into behaviour the dataset never covered, no feedback exists there at all. Iterative variants regenerate candidates and gather new pairs, which restores a data-production loop even though it still avoids PPO’s machinery.

The mechanical details deserve the same care as the mathematics. Padding must be masked. Prompt tokens must be excluded consistently on both sides. Truncation must never delete the evidence that determined the label — a rejected response cut off before its error is a rejected response with no visible reason to be rejected. And $\beta$ conventions differ across libraries, so a value copied from someone else’s config may mean something else in yours.

The durable model is pairwise policy shaping. DPO makes the desired response more characteristic of the new policy relative to a fixed baseline, and the rejected response less so. It trades a learned scalar judge and an online optimizer for an elegant offline loss — while the human judgments, the reference choice, the template branch, and the evaluation remain the real specification.