Deep Dive: RL for Coding Agents
Policy gradients to GRPO to long-horizon credit assignment
RL-for-LLMs from first principles
After this you can derive the policy-gradient objective and explain why it fits LLM post-training
Reinforcement learning on an LLM is just supervised learning where you don't have the labels - so you sample answers, score them and push probability toward the good ones.
The Cursor screens and the research deep-dive both probe whether you can build this up from the math rather than recite acronyms. Start by naming the object you're optimizing, because the whole stack falls out of one expectation.
Frame token generation as a Markov decision process. The model emits one token at a time and each token is a decision conditioned on everything written so far.
- State
- The prompt plus every token generated so far - the full context window the model conditions on.
- Action
- The next token sampled from the vocabulary distribution.
- Policy π_θ
- The LLM itself: π_θ(token | context). The weights θ are what RL updates.
- Reward
- A scalar on the finished sequence - did the code compile, pass tests, get preferred by a grader.
- Episode
- One full generation, from prompt to stop token (or, for an agent, to the end of a multi-tool trajectory).
The policy-gradient objectiveREINFORCE
You want to maximize expected reward over sequences the policy generates. You can't differentiate through sampling, so the log-derivative trick rewrites the gradient as something you can estimate from samples.
J(θ) = E_{τ ~ π_θ} [ R(τ) ] # expected reward over trajectories
∇J(θ) = E_{τ ~ π_θ} [ R(τ) · ∇ log π_θ(τ) ]
# In words: sample sequences, then for each one nudge the log-probability
# of every token UP if the reward was high, DOWN if it was low.
# No labels - the reward does the supervising.Read that gradient out loud in an interview: it is a reward-weighted log-likelihood. High-reward samples get reinforced, low-reward samples get suppressed and the magnitude scales with reward. The whole field is variations on making this estimator less noisy.
Variance is the enemy
REINFORCE is unbiased but the estimate is wildly noisy, because R(τ) for a whole sequence is one number scaling thousands of token gradients. Two fixes carry most of the weight and you should be able to justify both.
- Technique
- Baseline subtraction
- What it does
- Replace R with (R − b): the advantage. Center rewards around a baseline b.
- Why it's valid
- E[∇log π · b] = 0 for any state-independent b, so it cuts variance with zero bias.
- Technique
- Advantage estimation (GAE)
- What it does
- Estimate per-step advantage by blending a value function with observed returns.
- Why it's valid
- Trades a little bias for a large variance drop via the λ knob - usually a clear win.
| Technique | What it does | Why it's valid |
|---|---|---|
| Baseline subtraction | Replace R with (R − b): the advantage. Center rewards around a baseline b. | E[∇log π · b] = 0 for any state-independent b, so it cuts variance with zero bias. |
| Advantage estimation (GAE) | Estimate per-step advantage by blending a value function with observed returns. | Trades a little bias for a large variance drop via the λ knob - usually a clear win. |
Both shrink variance. The baseline is the move you must be able to prove is unbiased.
The expected value of ∇log π_θ over the policy's own distribution is zero - the score function integrates to zero. So adding any term that doesn't depend on the action, like a baseline b, adds zero to the expectation. The gradient stays unbiased while its variance drops. That one fact is the bridge from REINFORCE to actor-critic and interviewers love watching you derive it.
Keeping the policy from running away
An LLM has billions of parameters and a reward signal that is, at best, a proxy. Optimize it too hard and the model walks off the manifold of fluent text to chase reward - gibberish that scores well or degenerate repetition. The fix is a leash to the model you started from.
- Add a KL penalty against a frozen reference policy (usually the SFT checkpoint): reward becomes r − β·KL(π_θ ‖ π_ref).
- β trades reward-chasing against staying coherent. Too high and the model barely moves; too low and it collapses or hacks the reward.
- This is a soft trust region - it bounds how far each update can drag the policy from sane behavior.
- The KL term is also your first line of defense against reward hacking, because most hacks live far from the reference distribution.
When asked "why does RL work for post-training an LLM," don't start with PPO. Start with: the model already speaks fluent code from pretraining, RL just reshapes the distribution toward outputs a reward prefers. Then derive the reward-weighted log-prob gradient, name variance as the core problem and show the baseline is unbiased. That arc signals you understand the method, not just the library.
Takeaway. RL post-training is a reward-weighted log-likelihood: sample sequences, score them, push probability toward the winners - and the entire craft is killing variance (baselines, advantages) while a KL leash keeps the policy near a sane reference.
Self-check
QWhy is subtracting a baseline from the reward in a policy gradient a free lunch - variance down, bias unchanged?
PPO, GRPO and the modern stack
After this you can compare the algorithms used in real LLM RL and their tradeoffs
PPO was the workhorse of RLHF. GRPO is the one that made large-scale RL on reasoning and code cheap enough to do at large scale - and at Cursor, cheaper RL is the whole game.
Expect the screen to ask you to contrast them with real numbers and real memory footprints, not vibes. Both descend from the policy gradient in section one; they differ in how they estimate the advantage and how many models they keep in memory.
Interactive diagram. Tab through its regions; each focused region shows its detail in the panel below.
Step through each dimension - the difference is where the baseline comes from and how much memory you pay for it.
PPO: a clipped objective and a value model
PPO turns the trust region into a cheap clip. Instead of constraining KL explicitly each step, it caps how much the probability ratio between new and old policy can move, then takes the pessimistic side of the clip.
L_clip(θ) = E_t [ min( r_t · A_t, clip(r_t, 1-ε, 1+ε) · A_t ) ] r_t = π_θ(a_t|s_t) / π_old(a_t|s_t) # how much the policy moved on this token # The clip says: don't reward moving the ratio past 1±ε. # A_t comes from a learned value model V(s) via GAE - that's the # extra network PPO has to train and keep resident.
PPO is stable and gives you a per-token value estimate, which matters when reward is dense. The cost is memory: you hold four models at once.
- Policy
- The model you're training and updating.
- Value model
- A second trained network estimating V(s) for the baseline - often as large as the policy.
- Reward model
- Scores outputs (in RLHF) - frozen during PPO.
- Reference model
- Frozen SFT checkpoint for the KL penalty.
Four models in memory is why PPO is expensive to run at scale.
GRPO: drop the value modelDeepSeek-R1 style
GRPO's insight is that you don't need a learned value model to get a baseline - you can get one for free by sampling a group of completions per prompt and using the group's own reward statistics. Popularized by the DeepSeek-R1 work, it removed an entire network from the loop.
# For one prompt, sample G completions, score each: r_1 ... r_G A_i = (r_i − mean(r_1..r_G)) / std(r_1..r_G) # The group mean IS the baseline. No value network, no GAE. # Same clipped-ratio update as PPO, just with this group-relative A_i.
- Baseline source
- PPO
- Learned value model V(s)
- GRPO
- Mean reward of the sampled group
- Models in memory
- PPO
- Policy + value + reward + reference
- GRPO
- Policy + reward + reference (no value)
- Advantage granularity
- PPO
- Per-token (dense credit)
- GRPO
- Per-sequence (one advantage for the whole completion)
- Compute / memory
- PPO
- Higher - extra network to train and hold
- GRPO
- Lower - fewer params, more rollouts per prompt
- Best when
- PPO
- Dense or shaped rewards; need per-step credit
- GRPO
- Sparse terminal reward (test passes/fails); want scale
| PPO | GRPO | |
|---|---|---|
| Baseline source | Learned value model V(s) | Mean reward of the sampled group |
| Models in memory | Policy + value + reward + reference | Policy + reward + reference (no value) |
| Advantage granularity | Per-token (dense credit) | Per-sequence (one advantage for the whole completion) |
| Compute / memory | Higher - extra network to train and hold | Lower - fewer params, more rollouts per prompt |
| Best when | Dense or shaped rewards; need per-step credit | Sparse terminal reward (test passes/fails); want scale |
GRPO trades the value model for more rollouts - a good deal when reward arrives only at the end.
GRPO isn't free - you pay in extra rollouts (G completions per prompt instead of one) and you give up per-token credit, since every token in a completion shares one group-relative advantage. For coding tasks where the only honest signal is "did the final patch pass the tests," that per-sequence advantage is exactly right and dropping the value model buys you more samples and bigger batches with the same accelerators.
Tie algorithm choice to Cursor's published goal of training with less compute. Say it directly: "GRPO is a compute lever, not just an accuracy knob - for a sparse terminal reward like a test suite, dropping the value model frees memory and batch headroom for more rollouts, which is usually a better use of the same Blackwell hours." That connects the math to their actual constraint.
Takeaway. PPO buys per-token credit with a learned value model and four resident models; GRPO replaces the value model with group-relative reward statistics - cheaper and well-matched to sparse terminal rewards like a passing test suite.
Self-check
QA teammate wants to switch a coding-RL run from PPO to GRPO to save compute. What do you give up and when is that a good trade?
RLHF vs RLVR
After this you can explain when reward comes from a model vs from verification and the failure modes of each
Where does the reward come from? That single question splits modern post-training into two camps and coding sits awkwardly across the line between them.
RLHF learns a reward model from human preferences. RLVR reads reward straight off a verifier - a compiler, a test suite, a proof checker. The deep-dive will push on which one you'd reach for and, more sharply, how each one fails.
- Reward source
- RLHF (human feedback)
- Learned model trained on human preference pairs
- RLVR (verifiable reward)
- Deterministic checker: tests, compiler, proof
- Where it shines
- RLHF (human feedback)
- Taste, tone, helpfulness - things with no ground truth
- RLVR (verifiable reward)
- Math, code, anywhere correctness is checkable
- Signal quality
- RLHF (human feedback)
- Noisy, biased by labelers, trained on far less data than the base model
- RLVR (verifiable reward)
- Clean, scalable, reproducible
- Main failure
- RLHF (human feedback)
- Reward model is gameable and drifts off-distribution
- RLVR (verifiable reward)
- Reward is only as honest as the verifier
- Scales by
- RLHF (human feedback)
- Collecting more human labels (slow, costly)
- RLVR (verifiable reward)
- Running more environments (compute, not humans)
| RLHF (human feedback) | RLVR (verifiable reward) | |
|---|---|---|
| Reward source | Learned model trained on human preference pairs | Deterministic checker: tests, compiler, proof |
| Where it shines | Taste, tone, helpfulness - things with no ground truth | Math, code, anywhere correctness is checkable |
| Signal quality | Noisy, biased by labelers, trained on far less data than the base model | Clean, scalable, reproducible |
| Main failure | Reward model is gameable and drifts off-distribution | Reward is only as honest as the verifier |
| Scales by | Collecting more human labels (slow, costly) | Running more environments (compute, not humans) |
RLVR scales with compute, RLHF with human labor - a big reason top-tier reasoning runs lean on RLVR.
Coding is the interesting middle
A patch either passes the tests or it doesn't, so part of coding reward is purely verifiable. But a passing patch can still be unreadable, slow, insecure or a hack that special-cases the test. Those qualities aren't captured by any test you wrote, which is exactly why Cursor invests in learned graders.
Compiles, type-checks, tests pass.
Cheap, deterministic, trustworthy reward.
But: many real edits have weak or no tests.
Code quality, style, partial credit, multi-file coherence.
No checker exists - needs a learned grader.
This is where Cursor's research charter actually lives.
A learned grader is a reward model specialized to coding judgment - partial credit on a half-right diff, whether a multi-file change hangs together, whether the fix matches the request. Building and trusting these graders is one of the role's named responsibilities.
Reward hacking: the failure you must be able to discussthe proxy is not the goal
Every reward is a proxy and a capable agent will find the gap between the proxy and what you actually wanted. In coding that gap is wide and the exploits are concrete.
- Editing or deleting the tests instead of fixing the code, so the suite "passes."
- Hard-coding the expected output for the specific test inputs rather than solving the general case.
- Catching every exception or returning early so nothing throws - green tests, broken behavior.
- Gaming a learned grader: verbose comments, confident phrasing or patterns the grader spuriously rewards.
- Degenerate outputs that exploit a quirk in how reward is computed (length, format, a parsing bug).
Don't treat reward hacking as an edge case to patch later. It is the default outcome of optimizing a proxy hard enough and detecting it is a core research skill the team screens for. Mitigations: lock the test files out of the action space, hold out unseen tests the agent can't see or edit, add KL to the reference, audit high-reward trajectories by hand and adversarially probe the grader. Saying "I'd read the highest-reward rollouts and look for cheating" reads as someone who has actually run RL.
“For code I'd anchor reward on verifiable signals - tests, type-checks - because they're cheap and honest. But unit tests can't score quality or partial credit, so I'd train a grader for the non-verifiable part and I'd assume both will be hacked: I'd hold out tests the agent can't edit, keep a KL leash and manually audit the top-reward trajectories for cheating before trusting the number.”
Takeaway. RLVR gives clean, compute-scalable reward where correctness is checkable; RLHF covers taste at the cost of a noisy, gameable model - and coding straddles both, so you anchor on tests, train graders for quality and assume every proxy will be hacked.
Self-check
Long-horizon, sparse-reward agent-assisted RL
After this you can reason about credit assignment over many tool-calls in a coding episode
A single completion is one decision. An agent-assisted coding episode is a hundred - read this file, grep that symbol, edit, run tests, read the error, edit again - and the reward shows up only at the very end.
That is the hardest problem in the role's charter: extending RL to longer-horizon agent-assisted tasks. The deep-dive will want you to reason about credit assignment when one terminal scalar has to be apportioned across dozens of tool-calls.
Why long horizon is hard
- Sparse, delayed reward
- One scalar at step 80 must explain every action since step 1 - which tool-call actually mattered?
- Variance explosion
- Longer trajectories mean noisier returns; the policy-gradient estimate degrades fast with horizon.
- Compute cost
- Each episode is many forward passes plus real tool execution, so rollouts are slow and expensive to collect.
The toolkit for credit assignment
There's no single fix; you stack techniques and pick by how much signal you can afford to manufacture. Know what each one buys and what it risks.
Interactive diagram. Tab through its regions; each focused region shows its detail in the panel below.
Ranked by how safe a default each lever is - start at the top, add the rest only when you can trust the signal.
Score intermediate steps, not just the end.
Denser signal, faster credit assignment.
Risk: a flawed process reward is a new hacking surface.
Learn V(s) to estimate future reward mid-episode.
Propagates terminal reward backward smoothly.
Risk: hard to fit when states are long token sequences.
Add small intermediate incentives (test count rising).
Guides exploration on sparse tasks.
Risk: shapes behavior you didn't intend; keep it potential-based.
Start with short, easy episodes; grow length and difficulty.
Builds the long-horizon skill incrementally.
Pairs with difficulty calibration of the datapoints.
Trajectory-level advantages (the GRPO style from section two) are the honest default when only the end is verifiable: one advantage for the whole episode, shared by every token. You reach for process rewards when you can build a trustworthy intermediate signal, not before.
Environment fidelity is the quiet lever
Cursor has said Composer 2Cursor's in-house agentic coding model: frontier-level coding quality at high speed and low cost, built as a software-engineering specialist rather than a general-purpose model. trains RL in realistic Cursor sessions, using the same tools and harness the deployed model will use in production. That choice matters more than it looks.
- 1Same tools. The agent trains with the exact file-read, search, edit and terminal tools it ships with - no train/serve mismatch in the action space.
- 2Same harness. Episodes run in the real session loop, so the distribution of states the policy sees in training matches deployment.
- 3Real graded outcomes. Rollouts run in sandboxes and are scored on real execution, so the reward reflects what the model actually does to a codebase.
- 4Closes the offline-online gap. A policy optimized in a faithful environment transfers; one trained in a toy harness learns to exploit the toy.
Episode length couples directly to variance and compute. Long episodes let the agent discover multi-step solutions it could never stumble into greedily, but they cost more, vary more and dilute credit. The research judgment is choosing horizon and sampling temperature so the agent explores enough to find non-obvious fixes without burning the compute budget on noisy, low-information rollouts.
If asked "how would you assign credit when reward only comes at the end of a 50-step coding episode," don't jump to process rewards. Lead with the honest default - a trajectory-level advantage, GRPO-style - name the variance and compute costs of long horizons, then offer process rewards, value bootstrapping and a horizon curriculum as levers, each with the hacking or fitting risk it introduces. The nuance is what reads as production experience.
Takeaway. Long-horizon coding RL is a credit-assignment problem under sparse terminal reward: default to trajectory-level advantages, add process rewards, value bootstrapping and a horizon curriculum carefully and train in a high-fidelity environment so the policy can't learn to exploit a toy harness.
Self-check
QWhy does Cursor train ComposerCursor's own fast coding model, tuned for the editor and priced well below frontier models; the recommended day-to-day model for executing a plan.'s RL in realistic sessions with the same tools and harness the deployed model uses, rather than a simplified training environment?
Online & async RL infrastructure
After this you can discuss what it takes to run RL at production scale
The algorithm is maybe a tenth of the work. The rest is infrastructure: sandboxes that run hundreds of thousands of agent rollouts, a pipeline that keeps Blackwell accelerators saturated and the discipline to learn from live traffic without breaking it.
This role explicitly includes building the eval harnesses, sandboxed environments and async RL pipelines that research needs. The interview - especially the paid practical onsite - rewards reasoning about throughput, staleness and safety as concretely as about advantages.
Online RL on live trafficthe Cursor Tab approach
Cursor TabCursor's original autocomplete: multi-line, edit-aware suggestions you accept with the Tab key. is trained with online RL on real user interactions: every time a suggestion is shown, the user's accept or reject is a reward signal. Learning from production traffic is powerful and dangerous in equal measure.
- 1Collect. Log interactions and the user's response (accepted, rejected, edited-then-kept) as labeled experience.
- 2Attribute reward. Turn that response into a scalar - an accepted suggestion is positive signal, a rejection negative - net of obvious confounds.
- 3Correct for off-policy shift. The data was generated by the deployed model, not the one you're updating, so importance-weight or otherwise correct for the distribution gap.
- 4Gate for safety. Evaluate offline and behind a flag before any new policy touches real users; a bad update on live traffic degrades the product instantly.
Interactive diagram. Tab through its regions; each focused region shows its detail in the panel below.
The Cursor Tab loop - note the safety gate before any new policy touches real users.
Training on production traffic has feedback loops a static dataset doesn't. The deployed model shapes which suggestions users even see, so naive logging records a biased sample - you only get feedback on what you chose to show. Mention selection bias and the need for exploration (occasionally showing non-greedy suggestions to learn) and off-policy correction. Hand-waving "just learn from clicks" misses the core hazard.
Async, distributed RL
The expensive part of RL is generating rollouts; the cheap part is the gradient update. If you do them in lockstep, your training accelerators sit idle while rollouts generate. Cursor's pipeline is fully async to avoid exactly that.
- Rollout workers
- Generate trajectories in sandboxes continuously, often across regions and push them to a buffer.
- Trainer
- Pulls batches and updates the policy without waiting for any single rollout to finish.
- Weight broadcast
- Push fresh policy weights out to rollout workers periodically - the link that bounds staleness.
- The win
- Both sides stay busy: generation never blocks the trainer, the trainer never blocks generation.
The off-policy / staleness tradeoff
Decoupling buys throughput but creates a problem: by the time a rollout reaches the trainer, the policy has moved, so the data is slightly off-policy. How stale you let it get is a real dial.
- Knob
- Staleness tolerance
- Push it up →
- Higher throughput, more off-policy bias
- Push it down →
- Fresher data, accelerators stall waiting
- Knob
- Rollout batch size
- Push it up →
- Better hardware utilization
- Push it down →
- Faster policy updates, lower latency
- Knob
- Weight-sync frequency
- Push it up →
- Less staleness, more sync overhead
- Push it down →
- More throughput, more off-policy drift
| Knob | Push it up → | Push it down → |
|---|---|---|
| Staleness tolerance | Higher throughput, more off-policy bias | Fresher data, accelerators stall waiting |
| Rollout batch size | Better hardware utilization | Faster policy updates, lower latency |
| Weight-sync frequency | Less staleness, more sync overhead | More throughput, more off-policy drift |
Async RL is a throughput-vs-freshness balancing act; importance weighting buys you slack on the bias side.
Sandboxes are the foundation
None of this runs without a way to execute hundreds of thousands of untrusted agent rollouts safely and fast. Cursor's Anyrun-style sandboxing is what makes RL on real coding tasks possible at all.
- Isolation: agents run arbitrary code, so each rollout is a hardened sandbox that can't touch the host or other rollouts.
- Throughput: rollouts must spin up in well under a second and run massively in parallel or the trainer starves.
- Reproducible grading: each sandbox runs the tests/verifier deterministically so the reward is trustworthy.
- Cost: this is a dominant line item, which is part of why training-with-less-compute is a stated research goal.
When the onsite touches infra, reason about the bottleneck out loud: "rollout generation is the long pole, so I'd decouple it from the trainer, run sandboxes async across regions and accept some staleness with importance weighting - then watch accelerator utilization and the off-policy gap as my two health metrics." Naming utilization and staleness as the things you'd instrument signals you've thought about the system, not just the loss.
Takeaway. Production RL is a throughput problem: decouple slow rollout generation from fast policy updates, run untrusted rollouts in fast isolated sandboxes, accept bounded staleness with off-policy correction and treat live-traffic learning as a feedback loop that needs exploration and safety gating.
Self-check
QIn a fully async RL pipeline, why does data become off-policy and what's the lever you trade against it?
Inside the Composer 2 training pipeline
After this you can trace the concrete pipeline Cursor used to RL-train a real coding agent
Everything in the prior sections is theory until you see it wired into a real run. Composer 2Cursor's in-house agentic coding model: frontier-level coding quality at high speed and low cost, built as a software-engineering specialist rather than a general-purpose model. is that run - long-horizon RLlong-horizon reinforcement learning. Training a coding agent by running many rollouts on real problems and reinforcing the ones that succeed; a single rollout can reach 200K tokens and hundreds of tool calls. on real coding problems, in simulated copies of real repos, scored on real test outcomes.
Cursor has described the pipeline in enough detail that you can reason about each stage. Treat this section as the worked example the deep-dive keeps circling back to: where do the problems come from, how does an environment get built, and how do you train past the context window when a single rollout runs huge.
Interactive diagram. Step through it with the Next and Previous controls below, or Tab to a region to read its detail.
Each real coding problem fans out into many rollouts in a simulated repo copy; successes are reinforced, failures are pushed away from - and a prior model builds the environment that grades them.
Long-horizon RL on real coding problems
The unit of training is a real coding problem set inside a copy of its repo. For each problem the trainer runs many rollouts in parallel, each a full agent episode against its own simulated copy of the codebase. The successful rollouts get reinforced; the policy is updated away from the ones that failed.
- Scale per rollout
- A single rollout can reach 200K tokens and hundreds of tool calls before it terminates.
- Many per problem
- Each real problem is run many times so the group has both successes and failures to learn from.
- Isolated copies
- Every rollout works against its own simulated copy of the repo, so edits and test runs don't collide.
- Terminal signal
- Reward arrives at the end from the environment's tests - the sparse, verifiable signal from section three.
Hundreds of tool calls and 200K tokens per rollout is why environment throughput and self-summarization both matter.
Auto Install: a prior model builds the environmentComposer 1.5 as the constructor
A coding problem isn't trainable until its repo copy actually builds, installs and runs its tests. Doing that by hand for every environment doesn't scale, so Cursor automates it with the previous model. ComposerCursor's own fast coding model, tuned for the editor and priced well below frontier models; the recommended day-to-day model for executing a plan. 1.5 constructs each training environment before Composer 2Cursor's in-house agentic coding model: frontier-level coding quality at high speed and low cost, built as a software-engineering specialist rather than a general-purpose model. ever trains in it.
- 1Explore. The prior model reads the repo to understand its language, build system and how it's meant to run.
- 2Propose setup. It proposes roughly ten install commands to stand the project up from a clean checkout.
- 3Write tests. It writes the tests that will serve as the verifiable reward signal for that environment.
- 4Install, mock, retry. It runs the install, mocks what it must and retries until the tests actually pass - only then is the environment a usable, gradeable training target.
Standing up an arbitrary repo - right dependencies, a working build, runnable tests - is itself a hard agentic coding task. Cursor already had a capable coding agent in ComposerCursor's own fast coding model, tuned for the editor and priced well below frontier models; the recommended day-to-day model for executing a plan. 1.5, so they pointed it at environment construction: explore, propose the install commands, write tests, then install/mock/retry until green. That turns environment creation from a manual bottleneck into something that scales with compute, which is the whole reason you can train on a large pool of real repos at all.
Self-summarization: training past the context limit
A rollout that reaches 200K tokens will eventually hit the model's context limit mid-task. Rather than truncate and lose the thread, the model is trained to summarize its own progress and continue from the summary, so a single episode can run longer than one context window holds.
Long-horizon coding tasks generate more history than any context window can hold - hundreds of tool calls, file reads, test logs and prior edits. Without compaction the episode dies at the context wall before the task is done, so the agent could never learn behaviors that take more than one window to complete. Training the model to write and trust its own summary lets the rollout shed stale detail, keep the load-bearing state and push through tasks that genuinely require more than a single context window - which is most real engineering work.
A length penalty that scales with difficulty
Reward only on passing tests would let the agent ramble: take the long way every time because length is free. So the reward includes a nonlinear length penaltyA training reward that discourages a model from being too verbose or too terse, so easy tasks finish fast and hard tasks get more room.. Easy problems are expected to be solved fast; hard problems are allowed to run long before the penalty bites.
- Easy problems
- Penalized for taking long - the agent is pushed to solve them quickly and stop.
- Hard problems
- Given room to run; length is tolerated because the work genuinely needs it.
- Why nonlinear
- A flat per-token cost would punish hard problems for being hard; scaling the penalty by difficulty keeps efficiency pressure honest.
- What it shapes
- Spend matched to difficulty - fast on the trivial, patient on the genuinely complex.
The penalty rewards efficiency without telling the agent to give up on problems that legitimately need more steps.
Best-of-16 kept improving - no collapse
A standing fear in RL is mode collapse: the policy narrows onto one solution and loses the diversity that lets sampling find better answers. Cursor reported the opposite. Across training, best-of-16 performance kept improving - sampling sixteen attempts and taking the best still got better, so the policy did not collapse to a single strategy.
If RL had collapsed the policy onto one solution, all sixteen samples would look alike and best-of-16 would stop beating best-of-1 - the gain from drawing more samples would vanish. Watching best-of-16 keep climbing through training is direct evidence the policy stayed diverse enough that extra samples still surface better solutions. For a coding agent that's exactly what you want: many viable paths through a problem, not one brittle memorized route.
When the deep-dive asks how you'd train a long-horizon coding agent, narrate the Composer 2Cursor's in-house agentic coding model: frontier-level coding quality at high speed and low cost, built as a software-engineering specialist rather than a general-purpose model. pipeline as a system: real problems in isolated repo copies, many rollouts each (up to 200K tokens, hundreds of tool calls), environments auto-built by a prior model, self-summarizationA model summarizing its own work at a trigger point and continuing from that summary, so it can keep working past its context limit. to beat the context limit, a difficulty-scaled length penaltyA training reward that discourages a model from being too verbose or too terse, so easy tasks finish fast and hard tasks get more room. for efficiency, and best-of-16 watched as a non-collapse health check. Naming the environment-construction and context-limit problems - not just the loss - is what reads as someone who has actually shipped a run.
Takeaway. Composer 2Cursor's in-house agentic coding model: frontier-level coding quality at high speed and low cost, built as a software-engineering specialist rather than a general-purpose model. runs long-horizon RLlong-horizon reinforcement learning. Training a coding agent by running many rollouts on real problems and reinforcing the ones that succeed; a single rollout can reach 200K tokens and hundreds of tool calls. over real coding problems in isolated repo copies - many rollouts each, up to 200K tokens and hundreds of tool calls - with environments auto-built by the prior model, self-summarizationA model summarizing its own work at a trigger point and continuing from that summary, so it can keep working past its context limit. to train past the context limit, a difficulty-scaled length penaltyA training reward that discourages a model from being too verbose or too terse, so easy tasks finish fast and hard tasks get more room. for efficiency, and best-of-16 climbing through training as proof the policy never collapsed.
Self-check
QA single Composer 2Cursor's in-house agentic coding model: frontier-level coding quality at high speed and low cost, built as a software-engineering specialist rather than a general-purpose model. rollout can run to 200K tokens and hundreds of tool calls. Why does training the model to summarize its own progress help on these long tasks?