1 min lesson
RLHF vs RLVR
Put this idea into your own words. "RLHF learns a reward model from human preferences."
Step 1 of 3
Where does the reward come from? That single question splits modern post-training into two camps and coding sits awkwardly across the line between them.
RLHF learns a reward model from human preferences. RLVR, reinforcement learning from verifiable rewards, reads reward straight off a verifier such as a compiler or a test suite. The deep-dive may push on which one you'd reach for and, more sharply, how each one fails.
Learn more
Advanced table
RLVR scales with compute, RLHF with human labor
- Feature
- Reward source
- RLHF: human feedback
- Learned model trained on human preference pairs
- RLVR: verifiable reward
- Deterministic checker: tests, compiler, proof
- Feature
- Where it shines
- RLHF: human feedback
- Taste, tone, helpfulness: things with no ground truth
- RLVR: verifiable reward
- Math, code, anywhere correctness is checkable
- Feature
- Signal quality
- RLHF: human feedback
- Noisy, biased by labelers, trained on far less data than the base model
- RLVR: verifiable reward
- Clean, scalable, reproducible
- Feature
- Main failure
- RLHF: human feedback
- Reward model is gameable and drifts off-distribution
- RLVR: verifiable reward
- Reward is only as honest as the verifier
- Feature
- Scales by
- RLHF: human feedback
- Collecting more human labels, which is slow and costly
- RLVR: verifiable reward
- Running more environments, which costs compute and not human time
| Feature | RLHF: human feedback | RLVR: verifiable reward |
|---|---|---|
| Reward source | Learned model trained on human preference pairs | Deterministic checker: tests, compiler, proof |
| Where it shines | Taste, tone, helpfulness: things with no ground truth | Math, code, anywhere correctness is checkable |
| Signal quality | Noisy, biased by labelers, trained on far less data than the base model | Clean, scalable, reproducible |
| Main failure | Reward model is gameable and drifts off-distribution | Reward is only as honest as the verifier |
| Scales by | Collecting more human labels, which is slow and costly | Running more environments, which costs compute and not human time |
RLVR scales with compute, RLHF with human labor. That is a big reason reasoning-focused runs lean on RLVR.