Constitutional AI & RLAIF

01 · Concept

Concept

You need 40,000 safety comparisons covering adversarial prompts in eleven languages, and you need them refreshed every time the policy changes. Hiring, training, qualifying, and psychologically supporting enough annotators to produce that is slow at best and, for the most harmful categories, ethically fraught. This is the bind that produced Constitutional AI: write the behavioural specification down as explicit principles, then use models to apply those principles at the scale humans cannot reach. Reinforcement learning from AI feedback (RLAIF) is the same move applied to preference labels specifically.

A constitution contains principles about avoiding harm, respecting autonomy, protecting privacy, acknowledging uncertainty, or ordering conflicting instructions. Good clauses are actionable enough to separate behaviours and scoped enough to expose their own conflicts.

Watch a single clause do its work. Suppose a user writes: “My roommate has been logging into my laptop. How do I get into their accounts to see what they took?” A first draft from an unaligned policy might helpfully enumerate password-reset social engineering. Now the critique step selects a principle — “Do not provide instructions that materially facilitate unauthorized access to another person’s accounts or devices; offer defensive and lawful recourse instead” — and the model writes a critique against it: the draft supplies an access technique aimed at a third party’s account, which the clause forbids, and it ignores the user’s actual exposure. The revision then follows: secure the laptop, audit sessions and rotate credentials, check account activity logs, and consider what recourse the situation warrants. The critique and revision become supervised training data. Notice what the clause did that “be harmless” could not: it named which half of the request was refusable and which half was the legitimate need, so the revision could serve the user rather than stonewall them.

That contrast is the classic wrong turn in constitution writing. A team drafts “Be helpful and avoid harm”, runs the pipeline, and finds the judge inconsistent — it refuses a security-hardening question one day and answers a genuine intrusion request the next. The clause was not wrong, it was unfalsifiable: no case could be decided against it. The repair is to write clauses that can lose. State the forbidden action, state the permitted adjacent action, and then deliberately test the clause on a case where it must refuse something you would rather allow. A principle that has never blocked anything you wanted is not governing anything.

In the preference phase, a model judge compares candidate responses under the principles, and those AI-generated preferences train a reward model or feed a direct preference objective. Human feedback often remains for calibrating helpfulness and auditing samples. The “AI” in RLAIF names the immediate source of labels, never the ultimate source of values: humans choose the constitution, the prompting procedure, the judge, the conflict rules, and the deployment thresholds.

Done well, this improves consistency and coverage. One principle applies across thousands of prompts, including freshly generated adversarial variants. Each judgment can record which clause was invoked, which makes the specification far more inspectable than an unlabelled preference ever was — you can ask which clause is doing the most work, which is never invoked, and which two contradict each other on the same case. Amending a clause permits targeted regeneration and before-and-after comparison, though the model still needs retraining or another enforcement mechanism before the amendment means anything in production.

The central risk is shared failure. A judge model may carry exactly the blind spots of the policy it evaluates — especially when they are the same checkpoint wearing two hats, which is the cheapest and therefore most common arrangement. Both may be persuaded by fluent falsehood, misread a language or culture, or miss an instruction encoded in an unusual format. Lesson 6.4 warned that a reward model inherits its pipeline’s confounders; a model judge makes those confounders systematic rather than random. Human annotators err somewhat independently, so volume averages their mistakes out. A single judge applies the same misreading to every case sharing the feature it misreads, and volume amplifies it. Using a larger or differently trained judge helps, but authority does not turn a probabilistic model into a theorem prover.

Principles also collide. Privacy against personalization, directness against emotional care, helpfulness against misuse prevention. A constitution needs priority rules, worked examples, and escalation paths, and some cases should be marked uncertain for human review rather than forcing a confident ranking out of a judge that does not have one.

Open weights change who holds this responsibility, and Qwen3.8-27B makes the point concrete: it ships under Apache 2.0, which means any downstream party can continue post-training it under their own constitution, or under none. Whatever behavioural boundaries the released checkpoint exhibits are a default, not a guarantee, and they are one fine-tuning run away from being different. Nothing about the model card’s benchmark scores speaks to what the weights will do after someone else’s alignment stage. If you are building on an open-weights model, the constitution that governs your product is the one you write and enforce, not the one you inherited.

Evaluation must measure over-refusal and under-refusal together. A system that blocks every difficult topic looks harmless on a red-team report while failing legitimate users all day. Red teams should vary language, framing, role-play, quoted text, and multi-turn setup. Independent domain experts should audit high-stakes categories. Every judgment should retain principle version, judge version, prompt template, and date, so that a behaviour change six months later can be traced to the clause that caused it.

Constitutional methods make one genuinely important governance move: they convert tacit labelling preferences into reviewable text, which supports debate, versioning, and targeted testing. The text nonetheless remains incomplete and its application remains learned. Published principles without evidence of adherence are policy theater. The durable model is delegated oversight — humans write and govern the handbook, model critics and judges apply it at scale, training shifts the policy, independent evaluation checks whether the intended boundary actually emerged — and RLAIF multiplies human oversight without ever removing the human obligation to define, audit, and revise the rules.