Skip to lesson
Exit
Deep Dive: RL for Coding Agents1 / 2

2 min lesson

Environment fidelity is the quiet lever

Explain your answer to "Why does it matter that Composer's RL runs in realistic Cursor sessions rather than a simplified training environment?" Add one concrete detail from the lesson.

Step 1 of 2

Environment fidelity is the quiet lever

Cursor has described training Composer 2Cursor's own coding model, built on the open Kimi K2.5 base as a software-engineering specialist. Cursor calls it frontier-level at coding and more cost-efficient than comparable models; Composer 2.5 has since replaced it in Cursor's model list. Press Enter for the full definition. with RL in realistic Cursor sessions. Read that as a fidelity choice, one that matters more than it looks: the closer training is to deployment, the more of the gain transfers.

  1. 1Same tools. The action space in training matches the one the agent has when deployed, so there is no train/serve mismatch in which tools exist.
  2. 2Same harness. Episodes run in a session loop like the deployed one, so the states the policy sees in training match deployment.
  3. 3Real graded outcomes. Rollouts run in sandboxes and are scored on real execution, so the reward reflects what the model actually does to a codebase.
  4. 4Closes the offline-online gap. A policy optimized in a faithful environment transfers; one trained in a toy harness learns to exploit the toy.
Exploration vs exploitation, in coding terms

Episode length couples directly to variance and compute. Long episodes let the agent discover multi-step solutions it could never stumble into greedily, but they cost more, vary more and dilute credit. The research judgment is choosing horizon and sampling temperature so the agent explores enough to find non-obvious fixes without burning the compute budget on noisy, low-information rollouts.

Interview move

If asked "how would you assign credit when reward only comes at the end of a 50-step coding episode," don't jump to process rewards. Lead with the honest default, a trajectory-level advantage in the GRPO style. Name the variance and compute costs of long horizons, then offer process rewards, value bootstrapping and a horizon curriculum as levers, each with the hacking or fitting risk it introduces.