2 min lesson
GRPO: drop the value model
Work through this case: "A teammate wants to switch a coding-RL run from PPO to GRPO to save compute. What do you give up and when is that a good trade?" Say what you would do and why.
Step 1 of 2
GRPO: drop the value modelDeepSeek-R1 style
GRPO's insight is that you don't need a learned value model to get a baseline - you can get one for free by sampling a group of completions per prompt and using the group's own reward statistics. Popularized by the DeepSeek-R1 work, it removed an entire network from the loop.
# For one prompt, sample G completions, score each: r_1 ... r_G A_i = (r_i − mean(r_1..r_G)) / std(r_1..r_G) # The group mean IS the baseline. No value network, no GAE. # Same clipped-ratio update as PPO, just with this group-relative A_i.
GRPO isn't free - you pay in extra rollouts (G completions per prompt instead of one) and you give up per-token credit, since every token in a completion shares one group-relative advantage. For coding tasks where the only honest signal is "did the final patch pass the tests," that per-sequence advantage is exactly right and dropping the value model buys you more samples and bigger batches with the same accelerators.
Tie algorithm choice to Cursor's published goal of training with less compute. Say it directly: "GRPO is a compute lever, not just an accuracy knob - for a sparse terminal reward like a test suite, dropping the value model frees memory and batch headroom for more rollouts, which is usually a better use of the same accelerator hours." That connects the math to their actual constraint.