1 min lesson
Worked sketch - Drill 1, the non-verifiable grader
Design an online-RL setup for a Tab-style feature. What are the four pieces you must address and the chief risk of training on live traffic?
Step 1 of 2
Worked sketch - Drill 1, the non-verifiable grader
- Signal
- Blend a learned reward model with cheap verifiable checks (compiles, lints, partial test pass) so it isn't a single gameable scalar.
- Partial credit
- Score multi-file edits per-hunk against intent, not all-or-nothing, so near-misses get gradient.
- Anti-gaming
- Hold an adversarial set of reward-hacked stubs; require the grader to rank them below real fixes or it's rejected.
- Drift guard
- Retrain the reward model on fresh policy samples and keep a KL leash so the policy can't run off into the grader's blind spots.
Lead every design answer with the failure mode, then the mitigation. “The obvious risk is the policy learns to satisfy the grader's proxy instead of the task, so I'd hold an adversarial set and require the grader to rank hacks below real fixes.” Anticipating the hack unprompted is the single clearest signal of a researcher who has actually trained these systems, not just read about them.
Generic RL design reads like a textbook. Tie the online-RL drill to the published Tab approach and the grader drill to non-verifiable coding rewards and reference the constraints they actually face - async multi-region pipelines, sandboxed environments, training on real sessions. Grounded specifics separate a candidate who studied Cursor from one who studied RL.