Reward models & human preference data

01 · Concept

Concept

You have two answers to the same question. One is 90 words, correct, and slightly blunt. The other is 320 words, correct, opens with a summary, closes with three caveats, and buries one small hedge that is not quite true. Which should the model learn to produce? There is no token-level target you can write down that captures the answer, because both responses are fluent, both are on topic, and the difference lives in a judgment nobody encoded in the training text. Supervised fine-tuning cannot help here: it can only say “imitate this one”, which silently declares every wording difference in the other one to be an error.

Preference data captures the comparative judgment directly. A record contains a prompt $x$, a chosen response $y^{+}$, and a rejected response $y^{-}$. Candidates are shown without identifying clues such as model names, and presentation order is randomized. The rubric may cover correctness, relevance, clarity, safety, citation fidelity, and instruction following. Reviewers need an escape hatch — ties, invalid tasks, insufficient expertise — because forcing a preference converts honest uncertainty into confident false supervision.

A reward model maps a prompt and a response to a scalar $r_\phi(x,y)$, and is trained with a pairwise logistic objective:

Work one example through the arithmetic, because the shape of this loss explains most of what the scalar can and cannot mean. Suppose the model scores the chosen response at 1.2 and the rejected one at 0.4. The margin is 0.8, and $\sigma(0.8)\approx 0.69$, so the loss is $-\log(0.69)\approx 0.37$. Now add five to both scores — chosen 6.2, rejected 5.4. The margin is still 0.8, so the loss is still 0.37. The optimizer is entirely indifferent to the absolute level. Push the margin to zero and the loss becomes $\log 2 \approx 0.69$; invert the ordering to a margin of $-0.8$ and it rises to about 1.17. Every gradient this objective produces is a gradient about ordering.

Which leads straight to the classic wrong turn. A team ships the reward model as a quality gate: responses scoring above 3.0 pass, below 3.0 get regenerated. It works for a week, then a retrained reward model shifts every score up by two and the gate passes everything. The threshold was never meaningful — the loss above proves that no such threshold exists in the learned quantity. The correct use is comparative: score several candidates for the same prompt and take the best, or calibrate a decision rule against held-out human judgments and re-calibrate every time the model is retrained.

Data collection determines what the proxy means. Annotators disagree because a prompt is ambiguous, the rubric is incomplete, or values genuinely differ. Agreement statistics help, but a majority vote does not turn a contested preference into an objective fact. Expert domains need qualified reviewers; multilingual data needs native competence; safety judgments need escalation paths and protections for annotators exposed to harmful material.

Candidate generation matters as much as labelling. If every comparison pits an excellent answer against nonsense, the reward model learns easy signals and cannot separate two plausible responses. On-policy or periodically refreshed data samples candidates from the policy you are actually improving, exposing the boundary it currently finds confusing. Hard negatives, adversarial cases, and length-balanced pairs reduce shortcut learning.

Length is the canonical confounder. In one dataset detailed answers genuinely are better, so the model learns “longer is better” and then rewards padding everywhere else. Formatting, disclaimers, citation-shaped strings, and confident tone become the same kind of shortcut. Slice evaluations should test whether scores still track the rubric once these surface properties are controlled.

When a policy optimizes against the reward model, it searches for high-scoring outputs and therefore walks straight into regions the reward model never saw during training. Errors that were rare become attractive. This is Goodhart’s law in operational form: a measure that becomes a target stops being a good measure. Regularization, conservative updates, refreshed comparisons, and independent evaluation limit the damage without abolishing it.

Where does this leave the course’s specimen? Qwen3.8-27B ships as a single post-trained checkpoint with no technical report, so nothing can be said about whether or how a reward model featured in its production, and this course will not speculate. What is documented is the other kind of evidence: the model card reports outcome scores such as SWE-bench Pro 61.7 and GPQA Diamond 89.2, both vendor-reported and awaiting independent reproduction. Those numbers come from checkers, not from judges — and that distinction is exactly the fork in the road ahead. When an outcome can be verified programmatically, lesson 6.8 shows you can skip the learned judge entirely. When it cannot, a reward model is one way to turn human judgments into a reusable scalar signal; direct preference methods such as DPO instead optimize the pairs themselves without fitting a separate reward model. Both approaches still inherit the data-quality limits in this lesson.

Calibration questions remain even for a well-built judge. Held-out pairwise accuracy is necessary and nowhere near sufficient. Examine ties, subgroup disagreement, adversarial sensitivity, score distributions, and whether higher scores correlate with task success measured independently. A reward model can rank familiar pairs beautifully and still steer optimization somewhere useless. The durable picture is a learned judge trained from relative examples: its usefulness comes from scaling a costly human signal, and its danger comes from the same compression, because a nuanced conditional judgment has been flattened into one number, and every optimization step downstream has forgotten what that number left out.