ORPO, KTO, SimPO & the alignment zoo

01 · Concept

Concept

Your product logs 200,000 thumbs-up and thumbs-down events per month. Not pairs — individual verdicts on individual responses, most of them to prompts nobody else ever sent. DPO wants a chosen and a rejected response for the same prompt, so to use this data you would have to invent pairings, which means either discarding most of the events or manufacturing comparisons nobody made. This is the concrete situation that produced the alignment zoo: not a search for a better loss function, but a series of answers to “what feedback do I actually have, and what can I afford to hold in memory while I learn from it?”

Four questions organize every method in the zoo. What is the feedback unit — a pair, or a single labelled response? How is a whole response scored — summed log-probability, average log-probability, odds? What anchors the policy — a frozen reference model, a fixed margin, a supervised term, nothing? And what does the objective push toward — a widened margin, a target margin, a likelihood plus a penalty? Once those four are explicit, the acronyms stop being fashion and become design choices you can test.

ORPO (Odds Ratio Preference Optimization) folds two signals into one training stage. A supervised term raises the likelihood of the chosen response exactly as SFT would; a preference term compares the odds of the chosen and rejected responses under the policy and penalizes the rejected one. There is no frozen second model anywhere in the objective, which is what “reference-free” means, and “monolithic” refers to collapsing the usual SFT-then-align pipeline into a single stage. The design responds directly to the phenomenon lesson 6.6 examined: a pure preference loss can widen the margin while the chosen response’s own likelihood sinks. ORPO’s supervised term holds a floor under it. In exchange, the weight between the two terms becomes part of your behavioural specification — too much imitation blunts the preference, too much rejection pressure distorts the likelihood.

KTO (Kahneman-Tversky Optimization) changes the data interface instead of the anchor, and it is the answer to the 200,000 thumbs. It consumes examples labelled desirable or undesirable without requiring two responses to the same prompt. Its prospect-theory motivation matters practically: gains and losses are evaluated against a reference point and can be weighted asymmetrically, so a thumbs-down can count for more than a thumbs-up, which is often what you want from product feedback. But unpaired data is not unbiased data. Users choose what to rate, positive and negative events arrive from different prompt populations, and one person’s dislike concerns tone while another’s concerns a wrong number. Record propensity, source, locale, task, model version, and the feedback widget itself, or the objective will learn your selection mechanism rather than your users’ preferences.

SimPO removes the reference model and uses a scaled, length-normalized log-probability as its implicit reward:

The pairwise loss trains the chosen reward to exceed the rejected reward by a target margin $\gamma$. The scale $\beta$ controls the strength of reward differences; neither parameter is a reference-policy constraint. Work through what length normalization does with $\beta=1$, because it is not a cosmetic adjustment. Take two responses to the same prompt. Response A is 40 tokens with a summed log-probability of $-60$. Response B is 200 tokens with a summed log-probability of $-240$. Under summed scoring, A scores $-60$ and B scores $-240$, so A wins decisively. Under SimPO’s normalization, A averages $-1.5$ per token and B averages $-1.2$, so B wins. Same two responses, same model, opposite ranking — produced entirely by the choice of reduction. Neither is “correct”; they encode different beliefs about whether a long response should be penalized for its length. Which is why the interaction with truncation policy and desired verbosity has to be evaluated on your data rather than assumed from a paper.

Here is the classic wrong turn. Someone circulates a table showing SimPO beating DPO by four points on a chat benchmark, and the team adopts SimPO. Six weeks later it underperforms their old DPO run. What went wrong is that the table compared two recipes, not two objectives: different base checkpoints, different preference datasets, different candidate generators, different chat templates, different learning rates, different length filters. The objective was one of a dozen variables that moved. To actually choose, fix everything else — same base model, same pairs, same template, same compute budget, same evaluation — and change only the loss. That experiment is boring, unglamorous, and the only one that answers the question.

The memory argument that motivated several of these methods deserves a second look in light of lesson 6.6. Holding a frozen reference alongside a trainable policy is genuinely expensive when both are full 27B-class checkpoints at 54 GB apiece. But if you are already training LoRA adapters on a quantized backbone, the reference is a forward pass with the adapters switched off — one copy of the weights, no extra memory. In that setup, reference-free methods lose most of their headline advantage, and you should choose among them on behavioural grounds instead: does the objective hold a floor under chosen likelihood, does it consume the feedback you actually collect, does it normalize length the way your product needs?

So the decision starts with the data contract, not the leaderboard. Reliable pairs plus an affordable reference makes DPO a strong baseline. Only binary feedback makes KTO’s interface the one that fits. A hard memory ceiling with no adapter trick available makes ORPO or SimPO attractive. Then compare against plain SFT and against task-specific baselines on identical data, because a surprising fraction of preference-optimization wins evaporate against a well-tuned supervised run.

The durable lesson is not the acronyms. Map every objective onto feedback unit, response score, anchor, and optimization target, and the zoo turns from a parade of names into a small set of testable choices — most of which you can settle in an afternoon with your own data.