1 min lesson
A length penalty that scales with difficulty
Rebuild the parts of "A length penalty that scales with difficulty", then say why each one matters.
Step 1 of 2
A length penalty that scales with difficulty
Reward only on passing tests would let the agent ramble: take the long way every time because length is free. So the reward includes a nonlinear length penaltyA training reward that discourages a model from being too verbose or too terse, so easy tasks finish fast and hard tasks get more room. Press Enter for the full definition.. Easy problems are expected to be solved fast; hard problems are allowed to run long before the penalty bites.
- Easy problems
- Penalized for taking long - the agent is pushed to solve them quickly and stop.
- Hard problems
- Given room to run; length is tolerated because the work genuinely needs it.
- Why nonlinear
- A flat per-token cost would punish hard problems for being hard; scaling the penalty by difficulty keeps efficiency pressure honest.
- What it shapes
- Spend matched to difficulty - fast on the trivial, patient on the genuinely complex.
The penalty rewards efficiency without telling the agent to give up on problems that legitimately need more steps.
Learn more
Full explanation
Best-of-16 kept improving - no collapse
Best-of-16 kept improving - no collapse
A standing fear in RL is mode collapse: the policy narrows onto one solution and loses the diversity that lets sampling find better answers. Cursor reported the opposite. Across training, best-of-16 performance kept improving - sampling sixteen attempts and taking the best still got better, so the policy did not collapse to a single strategy.
If RL had collapsed the policy onto one solution, all sixteen samples would look alike and best-of-16 would stop beating best-of-1 - the gain from drawing more samples would vanish. Watching best-of-16 keep climbing through training is direct evidence the policy stayed diverse enough that extra samples still surface better solutions. For a coding agent that's exactly what you want: many viable paths through a problem, not one brittle memorized route.
When the deep-dive asks how you'd train a long-horizon coding agent, narrate the Composer 2Cursor's in-house agentic coding model: frontier-level coding quality at high speed and low cost, built as a software-engineering specialist rather than a general-purpose model. Press Enter for the full definition. pipeline as a system: real problems in isolated repo copies, many rollouts each (up to 200K tokens, hundreds of tool calls), environments auto-built by a prior model, self-summarizationA model summarizing its own work at a trigger point and continuing from that summary, so it can keep working past its context limit. Press Enter for the full definition. to beat the context limit, a difficulty-scaled length penaltyA training reward that discourages a model from being too verbose or too terse, so easy tasks finish fast and hard tasks get more room. Press Enter for the full definition. for efficiency, and best-of-16 watched as a non-collapse health check. Naming the environment-construction and context-limit problems - not just the loss - is what reads as someone who has actually shipped a run.
QA single Composer 2Cursor's in-house agentic coding model: frontier-level coding quality at high speed and low cost, built as a software-engineering specialist rather than a general-purpose model. Press Enter for the full definition. rollout can run to 200K tokens and hundreds of tool calls. Why does training the model to summarize its own progress help on these long tasks?