1 min lesson
Variance is the enemy
Use two rows in "Variance is the enemy" to state the practical decision rules.
Step 1 of 2
Variance is the enemy
REINFORCE is unbiased but the estimate is wildly noisy, because R(τ) for a whole sequence is one number scaling thousands of token gradients. Two fixes carry most of the weight and you should be able to justify both.
- Technique
- Baseline subtraction
- What it does
- Replace R with
R − b, the advantage. Center rewards around a baseline b. - Why it's valid
E[∇log π · b] = 0for any state-independent b, so it cuts variance with zero bias.
- Technique
- Generalized advantage estimation, or GAE
- What it does
- Estimate per-step advantage by blending a learned value function with observed returns.
- Why it's valid
- The λ knob slides between observed returns and the value estimate. It trades a little bias for a large variance drop, usually a clear win.
| Technique | What it does | Why it's valid |
|---|---|---|
| Baseline subtraction | Replace R with R − b, the advantage. Center rewards around a baseline b. | E[∇log π · b] = 0 for any state-independent b, so it cuts variance with zero bias. |
| Generalized advantage estimation, or GAE | Estimate per-step advantage by blending a learned value function with observed returns. | The λ knob slides between observed returns and the value estimate. It trades a little bias for a large variance drop, usually a clear win. |
Both shrink variance. The baseline is the move you must be able to prove is unbiased.
The expected value of ∇log π_θ over the policy's own distribution is zero, because the score function integrates to zero. So adding any term that doesn't depend on the action, like a baseline b, adds zero to the expectation. The gradient stays unbiased while its variance drops. That one fact is the bridge from REINFORCE to actor-critic, and it is a standard thing to be asked to derive.