1 min lesson
Keeping the policy from running away
Give a plain answer to "Why is subtracting a baseline from the reward in a policy gradient a free lunch: variance down, bias unchanged?" Then ground it in one lesson detail.
Step 1 of 2
Keeping the policy from running away
An LLM has billions of parameters and a reward signal that is, at best, a proxy. Optimize it too hard and the model walks off the manifold of fluent text to chase reward: gibberish that scores well, or degenerate repetition. The fix is a leash to the model you started from.
- Add a KL penalty, the Kullback-Leibler divergence between the policy and a frozen reference policy, usually the SFT checkpoint. Reward becomes
r − β·KL(π_θ ‖ π_ref). - β trades reward-chasing against staying coherent. Too high and the model barely moves; too low and it collapses or hacks the reward.
- This is a soft trust region. It bounds how far each update can drag the policy from sane behavior.
- The KL term is also your first line of defense against reward hacking, because most hacks live far from the reference distribution.
When asked "why does RL work for post-training an LLM," don't start with Proximal Policy Optimization, or PPO. Start with the model: it already speaks fluent code from pretraining, and RL reshapes the distribution toward outputs a reward prefers. Then derive the reward-weighted log-prob gradient, name variance as the core problem and show the baseline is unbiased.