A freshly pretrained LLM just predicts text — it isn't helpful, honest, or safe by default. RLHF is the process that turns that raw next-token predictor into a usable, aligned assistant.
The pipeline
1. Pretrain — learn language from the internet (knows everything, follows nothing)
2. SFT — fine-tune on example instructions + good answers (learns to follow)
3. Reward model — humans rank answers; train a model to predict their preference
4. RL — optimize the LLM to maximize that reward (PPO / DPO)The result follows instructions, adopts a helpful tone, and refuses harmful requests.
Why it matters
- It's how models learn to be helpful and where refusals come from.
- The reward model is a proxy for human values — so RLHF is a real-world alignment attempt, complete with reward-hacking risk (Goodhart's Law).
- DPO is a popular simpler alternative that skips the separate RL step.
RLHF aligns a model to the preferences of its raters — which raises the question of whose values get encoded.
Related: The Alignment Problem · Jailbreaking LLMs · Specification Gaming and Reward Hacking