2 min lesson
Async, distributed RL
Walk through each part of "Async, distributed RL", then explain what each one does.
Step 1 of 3
Async, distributed RL
The expensive part of RL is generating rollouts; the cheap part is the gradient update. If you do them in lockstep, your training accelerators sit idle while rollouts generate. Cursor's pipeline is fully async to avoid exactly that.
Decoupling rollouts from updates
- Rollout workers
- Generate trajectories in sandboxes continuously, often across regions and push them to a buffer.
- Trainer
- Pulls batches and updates the policy without waiting for any single rollout to finish.
- Weight broadcast
- Push fresh policy weights out to rollout workers periodically - the link that bounds staleness.
- The win
- Both sides stay busy: generation never blocks the trainer, the trainer never blocks generation.
Learn more
Full explanation
The off-policy / staleness tradeoff
The off-policy / staleness tradeoff
Decoupling buys throughput but creates a problem: by the time a rollout reaches the trainer, the policy has moved, so the data is slightly off-policy. How stale you let it get is a real dial.
- Knob
- Staleness tolerance
- Push it up →
- Higher throughput, more off-policy bias
- Push it down →
- Fresher data, accelerators stall waiting
- Knob
- Rollout batch size
- Push it up →
- Better hardware utilization
- Push it down →
- Faster policy updates, lower latency
- Knob
- Weight-sync frequency
- Push it up →
- Less staleness, more sync overhead
- Push it down →
- More throughput, more off-policy drift
| Knob | Push it up → | Push it down → |
|---|---|---|
| Staleness tolerance | Higher throughput, more off-policy bias | Fresher data, accelerators stall waiting |
| Rollout batch size | Better hardware utilization | Faster policy updates, lower latency |
| Weight-sync frequency | Less staleness, more sync overhead | More throughput, more off-policy drift |
Async RL is a throughput-vs-freshness balancing act; importance weighting buys you slack on the bias side.