Reinforcement Learning from Human Feedback (RLHF) is a post-training alignment approach where a language model (policy) is optimized using a reward model trained on human preference data. The policy is improved with reinforcement learning (commonly PPO) under a KL penalty to remain close to a reference model, steering outputs toward helpfulness, harmlessness, and desired style.
What is Reinforcement Learning from Human Feedback (RLHF)?
RLHF typically proceeds in stages: (1) Supervised fine-tuning (SFT) on instruction-following data to get a reasonable policy; (2) Collection of human preference pairs on model generations for the same prompt (chosen vs. rejected); (3) Training a reward model to score responses; (4) Policy optimization with PPO or similar algorithms, maximizing expected reward while constraining divergence from a frozen reference via a KL term. Practical recipes include response-length normalization, reward shaping, entropy bonuses, and prompt/response filtering. Data can be collected continuously (online) or batched (offline), with evaluators calibrated for consistency and bias control.
Why it matters and where it’s used
RLHF aligns open-ended behaviors that are hard to specify with rules, improving helpfulness, safety refusals, tone, and instruction adherence. It enables brand/style control, reduces toxic or private data disclosures, and supports long-horizon behaviors like multi-step reasoning or planning. Enterprises use RLHF to fine-tune assistants, domain copilots, red-team hardened models, and policies that balance creativity with compliance.
Examples
- Safety alignment: increase reward for safe refusals to disallowed queries.
- Brand/style tuning: reward succinct, courteous responses for customer support.
- Long-form summarization: reward grounded, cited summaries over verbose hallucinations.
- Code assistance: prefer compilable, tested snippets and correct fixes.
FAQs
- How is RLHF different from DPO? RLHF trains a reward model and uses RL (e.g., PPO). DPO optimizes directly on preference pairs without a reward model.
- Does RLHF guarantee safety? No. It improves averages but needs guardrails, tool policies, and continuous evaluation.
- What data do I need? High-quality prompt sets with multiple model completions and human rankings; clear rubrics and calibration are crucial.
- Is PPO required? PPO is common but not mandatory; other policy-gradient or KL-regularized objectives exist.
- Cost and complexity? Higher than SFT/DPO due to reward modeling, on-policy rollouts, and safety evaluation.
- How to evaluate? Use preference win-rate, safety/harm benchmarks, faithfulness, citation quality, and downstream task success.