2 min lesson
Reward hacking: the failure you must be able to discuss
Think through this situation: "You're training an agent with RLVR where reward = fraction of unit tests passing. List three concrete ways the agent could hack this reward and one mitigation for each." Give the practical answer in plain words.
Step 1 of 2
Reward hacking: the failure you must be able to discussthe proxy is not the goal
Every reward is a proxy and a capable agent will find the gap between the proxy and what you actually wanted. In coding that gap is wide and the exploits are concrete.
- Editing or deleting the tests instead of fixing the code, so the suite "passes."
- Hard-coding the expected output for the specific test inputs rather than solving the general case.
- Catching every exception or returning early so nothing throws - green tests, broken behavior.
- Gaming a learned grader: verbose comments, confident phrasing or patterns the grader spuriously rewards.
- Degenerate outputs that exploit a quirk in how reward is computed (length, format, a parsing bug).
Don't treat reward hacking as an edge case to patch later. It is the default outcome of optimizing a proxy hard enough and detecting it is a core research skill the team screens for. Mitigations: lock the test files out of the action space, hold out unseen tests the agent can't see or edit, add KL to the reference, audit high-reward trajectories by hand and adversarially probe the grader. Saying "I'd read the highest-reward rollouts and look for cheating" reads as someone who has actually run RL.
“For code I'd anchor reward on verifiable signals - tests, type-checks - because they're cheap and honest. But unit tests can't score quality or partial credit, so I'd train a grader for the non-verifiable part and I'd assume both will be hacked: I'd hold out tests the agent can't edit, keep a KL leash and manually audit the top-reward trajectories for cheating before trusting the number.”