1 min lesson
Long-horizon RL on real coding problems
Name the parts in "Long-horizon RL on real coding problems" and give the practical job of each one.
Step 1 of 2
Long-horizon RL on real coding problems
The unit of training is a real coding problem set inside a copy of its repo. For each problem the trainer runs many rollouts in parallel, each a full agent episode against its own simulated copy of the codebase. The successful rollouts get reinforced; the policy is updated away from the ones that failed.
What one rollout actually costs
- Scale per rollout
- A single rollout can reach 200K tokens and hundreds of tool calls before it terminates.
- Many per problem
- Each real problem is run many times so the group has both successes and failures to learn from.
- Isolated copies
- Every rollout works against its own simulated copy of the repo, so edits and test runs don't collide.
- Terminal signal
- Reward arrives at the end from the environment's tests - the sparse, verifiable signal from section three.
Hundreds of tool calls and 200K tokens per rollout is why environment throughput and self-summarization both matter.