Reward models & human preference data

1

Concept

Many qualities we want from an assistant are hard to express as a token-level target. Two answers can use different words and both be correct; a fluent answer can be subtly misleading; a refusal can be either necessary or overly cautious. Preference data captures a comparative judgment: for the same prompt, a reviewer sees candidate responses and chooses which better satisfies a rubric.

A typical record contains a prompt $x$, a chosen response $y^+$, and a rejected response $y^-$. Candidates should be presented without irrelevant clues such as model names, and order should be randomized. The rubric may cover correctness, relevance, clarity, safety, citation fidelity, and instruction following. Reviewers need an option for ties, invalid tasks, or insufficient expertise; forcing a preference can turn uncertainty into false supervision.

A reward model maps a prompt and response to a scalar $r_\phi(x,y)$. It is commonly trained with a pairwise logistic objective:

The loss cares about score differences, not an absolute unit of goodness. Adding the same constant to both scores changes nothing. After training, the model can score newly generated responses, providing a differentiable or efficiently queried proxy that a policy optimizer can maximize.

Data collection determines what this proxy means. Annotators can disagree because a prompt is ambiguous, the rubric is incomplete, or values genuinely differ. Agreement statistics are useful, but majority vote does not make a contested preference objective. Expert domains need qualified reviewers. Multilingual data needs native competence. Safety judgments need clear escalation and psychological protections for annotators exposed to harmful material.

Candidate generation matters too. If every comparison pits an excellent answer against nonsense, the reward model learns easy signals and may not distinguish two plausible responses. On-policy or periodically refreshed data samples candidates from the policy being improved, exposing the boundary it currently finds confusing. Hard negatives, adversarial examples, and balanced lengths can reduce shortcut learning.

Length is a classic confounder. Detailed answers can genuinely be better in one dataset, so a reward model may learn “longer is better” and reward unnecessary verbosity elsewhere. Formatting, disclaimers, citations, or confident tone can become similar shortcuts. Slice evaluations should test whether scores track the intended rubric after controlling for these superficial properties.

When a policy optimizes the reward model, it searches for outputs with high predicted reward. This creates distribution shift: the policy discovers regions the reward model did not see during training. Errors that were rare can become attractive. The phenomenon is often described through Goodhart’s law—once a measure becomes a target, it may stop being a good measure. Regularization, conservative updates, refreshed comparisons, and independent evaluations limit but do not abolish this risk.

Reward models also invite calibration questions. Pairwise accuracy on held-out comparisons is necessary but insufficient. Teams should examine ties, subgroup disagreement, adversarial sensitivity, score distributions, and whether higher scores correlate with task success measured independently. A reward model can rank familiar pairs well and still guide optimization badly.

The durable picture is a learned judge trained from relative examples. Its usefulness comes from scaling a costly human signal to many candidate outputs. Its danger comes from the same compression: a nuanced, conditional judgment becomes one scalar. Every later optimization step must remember what information that scalar omitted.