2 min lesson
Keeping the policy from running away
Give a plain answer to "Why is subtracting a baseline from the reward in a policy gradient a free lunch - variance down, bias unchanged?" Then ground it in one lesson detail.
Step 1 of 3
Keeping the policy from running away
An LLM has billions of parameters and a reward signal that is, at best, a proxy. Optimize it too hard and the model walks off the manifold of fluent text to chase reward - gibberish that scores well or degenerate repetition. The fix is a leash to the model you started from.
- Add a KL penalty against a frozen reference policy (usually the supervised fine-tuning, or SFT, checkpoint): reward becomes r − β·KL(π_θ ‖ π_ref).
- β trades reward-chasing against staying coherent. Too high and the model barely moves; too low and it collapses or hacks the reward.
- This is a soft trust region - it bounds how far each update can drag the policy from sane behavior.
- The KL term is also your first line of defense against reward hacking, because most hacks live far from the reference distribution.
When asked "why does RL work for post-training an LLM," don't start with PPO. Start with: the model already speaks fluent code from pretraining, RL just reshapes the distribution toward outputs a reward prefers. Then derive the reward-weighted log-prob gradient, name variance as the core problem and show the baseline is unbiased. That arc signals you understand the method, not just the library.