1 min lesson
Variance is the enemy
Use two rows in "Variance is the enemy" to state the practical decision rules.
Step 1 of 2
Variance is the enemy
REINFORCE is unbiased but the estimate is wildly noisy, because R(τ) for a whole sequence is one number scaling thousands of token gradients. Two fixes carry most of the weight and you should be able to justify both.
- Technique
- Baseline subtraction
- What it does
- Replace R with (R − b): the advantage. Center rewards around a baseline b.
- Why it's valid
- E[∇log π · b] = 0 for any state-independent b, so it cuts variance with zero bias.
- Technique
- Advantage estimation (GAE)
- What it does
- Estimate per-step advantage by blending a value function with observed returns.
- Why it's valid
- Trades a little bias for a large variance drop via the λ knob - usually a clear win.
| Technique | What it does | Why it's valid |
|---|---|---|
| Baseline subtraction | Replace R with (R − b): the advantage. Center rewards around a baseline b. | E[∇log π · b] = 0 for any state-independent b, so it cuts variance with zero bias. |
| Advantage estimation (GAE) | Estimate per-step advantage by blending a value function with observed returns. | Trades a little bias for a large variance drop via the λ knob - usually a clear win. |
Both shrink variance. The baseline is the move you must be able to prove is unbiased.
The expected value of ∇log π_θ over the policy's own distribution is zero - the score function integrates to zero. So adding any term that doesn't depend on the action, like a baseline b, adds zero to the expectation. The gradient stays unbiased while its variance drops. That one fact is the bridge from REINFORCE to actor-critic and interviewers love watching you derive it.