1 min lesson
PPO: a clipped objective and a value model
Use "PPO: a clipped objective and a value model" to explain each part and the role it plays.
Step 1 of 2
PPO: a clipped objective and a value model
PPO turns the trust region into a cheap clip. Instead of constraining KL explicitly each step, it caps how much the probability ratio between new and old policy can move, then takes the pessimistic side of the clip.
PPO clipped surrogate - r_t is the new/old prob ratio, A_t the advantage
L_clip(θ) = E_t [ min( r_t · A_t, clip(r_t, 1-ε, 1+ε) · A_t ) ] r_t = π_θ(a_t|s_t) / π_old(a_t|s_t) # how much the policy moved on this token # The clip says: don't reward moving the ratio past 1±ε. # A_t comes from a learned value model V(s) via GAE - that's the # extra network PPO has to train and keep resident.
PPO is stable and gives you a per-token value estimate, which matters when reward is dense. The cost is memory: you hold four models at once.
What PPO keeps resident
- Policy
- The model you're training and updating.
- Value model
- A second trained network estimating V(s) for the baseline - often as large as the policy.
- Reward model
- Scores outputs (in RLHF) - frozen during PPO.
- Reference model
- Frozen SFT checkpoint for the KL penalty.
Four models in memory is why PPO is expensive to run at scale.