1 min lesson
Calibration: does the grader rank truth above plausibility
Imagine this comes up at work: "You've built an LLM-as-judge grader and it gives sensible-looking scores. What's the single most important thing to verify before using it in a training run and how?" Start with the practical move.
Step 1 of 2
Calibration: does the grader rank truth above plausibilitythe only question that matters
A grader is useless if it can't rank a correct solution above a plausible-but-wrong one. That is the property you measure and it is measurable.
- 1Assemble gold pairs. Collect tasks where you trust the labels - known-correct solutions and known-bad ones, including subtle wrong answers, not just obvious garbage.
- 2Score agreement. Run the grader and measure how often it ranks the good solution above the bad one. Report pairwise accuracy or correlation with the human label, not just average score.
- 3Slice the failures. Where the grader disagrees with humans, look at the cases - verbose-but-wrong, terse-but-right, correct-but-unconventional. The pattern of disagreement is the grader's bias.
- 4Recalibrate. Tighten the rubric, add negative examples to the RM training set or change the gate threshold, then re-measure on a held-out slice.
Adversarially test before you trust
A grader that is easy to game produces a policy that games it. Before wiring a grader into a training run, attack it yourself: feed it padded answers, plausible-wrong solutions, outputs that flatter the rubric without solving the task. If you can find cheap reward without real capability, so will the policy - and it has millions of rollouts to search.