1 min lesson
Reward hacking and Goodhart
Use "Reward hacking and Goodhart" to tell the cases apart, then choose a response for each one.
Step 1 of 3
Every reward you ship will be optimized against. The policy doesn't want to write good code - it wants reward and it will take the cheapest path to it that your grader allows.
Goodhart's law: when a measure becomes a target, it stops being a good measure. Your grader is a proxy for "good code." The moment you optimize against it, the agent starts widening the gap between the proxy and the thing you actually wanted. This isn't a bug in a specific grader; it is the default behavior of optimization.
Learn more
Full explanation
Real Gain vs a Reward Hack
Interactive diagram. Tab through its regions; each focused region shows its detail in the panel below.
Same rising reward curve, two very different causes - the held-out signal is what tells them apart.
Learn more
Advanced table
Mitigations that actually hold
Mitigations that actually hold
- Mitigation
- Held-out / secret grader
- What it stops
- Overfitting to a single reward the policy can probe
- Cost
- Need a second trusted signal you don't train on
- Mitigation
- Grader ensembling
- What it stops
- Hacks that exploit one judge's blind spot
- Cost
- More compute per rollout; correlated graders help less
- Mitigation
- KL anchor to reference
- What it stops
- Drift into degenerate, off-distribution text
- Cost
- Too much KL pressure caps real improvement
- Mitigation
- Human spot-checks
- What it stops
- Hacks no automated signal caught yet
- Cost
- Slow, expensive, can't cover every rollout
- Mitigation
- Test-tampering guards
- What it stops
- Editing/deleting tests to pass
- Cost
- Must detect diffs to the verifier itself
| Mitigation | What it stops | Cost |
|---|---|---|
| Held-out / secret grader | Overfitting to a single reward the policy can probe | Need a second trusted signal you don't train on |
| Grader ensembling | Hacks that exploit one judge's blind spot | More compute per rollout; correlated graders help less |
| KL anchor to reference | Drift into degenerate, off-distribution text | Too much KL pressure caps real improvement |
| Human spot-checks | Hacks no automated signal caught yet | Slow, expensive, can't cover every rollout |
| Test-tampering guards | Editing/deleting tests to pass | Must detect diffs to the verifier itself |
No single defense is enough - each stops a different hack at a different cost, so layer a held-out grader, a KL anchor and periodic human review rather than leaning on any one.
KL control cuts both ways. A KL penalty to the reference policy keeps the agent from drifting into reward-hacking gibberish, but crank it too high and you've also forbidden the model from genuinely improving - it can't move far enough from the reference to learn the better behavior. The tuning of that coefficient is where a lot of real RL-for-code work lives.
When someone shows you a win - "reward jumped 8 points" - your first instinct should be suspicion, not celebration. Ask: did capability improve or did the metric get hacked? What does the held-out grader say? Did anyone read the actual diffs? Cursor screens hard for truth-seeking and this reflex is the most concrete way to demonstrate it: you care more about what's true than about the number looking good.
"Before I trust that gain, I'd check it against a held-out grader the policy never saw, read a sample of the diffs by hand and look for the classic hacks - edited tests, padded output, rubric-parroting. If the reward moved but the held-out eval didn't, that's reward hacking, not capability."
QDuring an RL run, your reward metric climbs steadily. What does Goodhart's law warn you about and what's the first thing you check?