2 min lesson
Parallelism and the RL loop bottleneck
Take this case and make the call: "Your agent-assisted RL training run is slow and GPU utilization on the learner is low. What's the most likely culprit and how would you confirm it before changing anything?"
Step 1 of 2
Parallelism and the RL loop bottleneckdata / tensor / pipeline / expert
- Data parallel
- Replicate the model, split the batch, all-reduce gradients. Simplest; bounded by gradient comms.
- Tensor parallel
- Split each matmul across GPUs. Heavy intra-layer comms; keep it inside a fast NVLink island.
- Pipeline parallel
- Split layers across GPUs in stages. Cheap comms but introduces bubbles you fight with micro-batching.
- Expert parallel
- Place MoE experts on different GPUs; tokens are routed there via all-to-all. The MoE-specific axis.
In RL the loop is generate rollouts, score them, then train on them - and the rollout (inference) phase often dominates wall-clock, because generation is sequential and the policy keeps changing. The async trick is to decouple actors from learners: rollout fleets generate against a slightly stale policy while learners update, instead of letting expensive GPUs idle between phases.
Actors and learners run on separate fleets and don't block each other.
Off-policy correction handles the lag between the rollout policy and the current policy.
Multi-region spreads the sandbox/GPU fleet to where capacity is.
Each coding episode runs in an isolated sandbox that can execute code and tests safely.
Must be fast to spin up and reproducible, since rollout throughput gates training.
Realistic Cursor sessions, not toy tasks, so reward reflects real edits.
When asked to speed up an RL run, profile the loop first and say so: "In agent-assisted RL the rollout phase usually dominates, so I'd measure the generate-vs-train split before touching kernels. If rollouts dominate, async actors and faster sandbox spin-up beat a tensor-parallel tweak." Profiling before optimizing is the truth-seeking habit they screen for.