Constitutional AI is an alignment framework where a model is trained to follow a written set of normative principles—a “constitution”—and uses those principles to critique and revise its own outputs. It reduces reliance on human labelers by replacing some preference data with AI‑guided self‑critique and revision, and can be combined with RLHF/DPO.
What is Constitutional AI?
The core recipe runs in two phases. First, supervised fine-tuning (SFT) teaches the model to produce helpful responses and to generate self‑critiques using a fixed principle list (e.g., be harmless, be honest, respect privacy, avoid discrimination). Next, the model applies those principles to critique its initial answer and produce a revised, more aligned answer; training prefers the revised answer. Variants extend this with Reinforcement Learning from AI Feedback (RLAIF), where a separate AI rater scores responses according to the same principles, or with DPO/ORPO that directly optimizes on chosen‑over‑rejected pairs generated via self‑revision. In deployment, the principles can be included in prompts to steer behavior without further training.
Why it matters and where it’s used
Constitutional AI scales alignment while making the rationale explicit and auditable. It reduces the volume of sensitive human labeling, speeds iteration on policy changes (edit principles, retrain), and improves safety refusals, tone, and disclosure practices. It’s used for general assistants, domain copilots, and safety-hardening pipelines that must reflect organizational or regulatory norms.
Examples
- Principle set: “Avoid illegal instructions,” “Respect privacy,” “Use citations for claims,” “Acknowledge uncertainty,” “Avoid hateful or biased content.”
- Self‑critique → revision: model critiques its answer for policy issues, then produces a corrected version.
- RLAIF: an AI judge scores outputs by the constitution; the policy is optimized to maximize that score.
- Policy updates: change a principle (e.g., citation requirements) and regenerate revised data for rapid post‑training.
FAQs
- How is this different from RLHF? RLHF learns from human preferences; Constitutional AI uses explicit principles for self‑critique/revision and can use AI feedback (RLAIF) to scale.
- Does it remove humans entirely? No. Humans still design/review principles, audit data, and evaluate downstream risks.
- Can it conflict with helpfulness? Principles trade off; use weighting or multi‑objective tuning and verify with evaluations.
- How do I choose principles? Start with safety, legality, privacy, honesty, respect, and application‑specific rules; keep them concise and testable.
- What are pitfalls? Over‑generic or contradictory principles, distribution shift, and models overfitting to phrasing; maintain evaluation suites and red‑team regularly.