GRPO: drop the value model

GRPO: drop the value modelDeepSeek-R1 style

GRPO's insight is that you don't need a learned value model to get a baseline. You can get one for free by sampling a group of completions per prompt and using the group's own reward statistics. Popularized by the DeepSeek-R1 work, it removed an entire network from the loop.

# For one prompt, sample G completions, score each: r_1 ... r_G
A_i = (r_i − mean(r_1..r_G)) / std(r_1..r_G)

# The group mean IS the baseline. No value network, no GAE.
# Same clipped-ratio update as PPO, just with this group-relative A_i.

# Worked example: G = 4 completions for one prompt, only one passes
rewards   r = [1, 0, 0, 0]
mean      = 0.25
std       = sqrt(0.1875) = 0.433        # population std: divide by G

A_pass    = (1 − 0.25) / 0.433 = +1.73
A_fail    = (0 − 0.25) / 0.433 = −0.58  # each of the three failures

# The advantages sum to zero within the group.

One pass in four gives the passing sample a large positive weight and each failure a small negative one, so the update pushes hard toward the one thing that worked. If all four pass or all four fail, every gap from the mean is zero, so the std is zero and the group carries no signal. A small epsilon in the denominator is the usual guard against dividing by zero, and difficulty calibration in section four exists to keep prompts out of those dead groups.