1 min lesson
Designing and training graders
Describe what "There is no single right grader" changes in practice.
Step 1 of 3
There is no single right grader. There is a menu of signals, each accurate in a different regime and the craft is choosing and combining them under a real throughput budget.
Learn more
Full explanation
The Gate-Then-Judge Grader Pipeline
Interactive diagram. Step through it with the Next and Previous controls below, or Tab to a region to read its detail.
The common production shape: cheap checks gate first, expensive judgment scores the survivors and you calibrate the whole thing before you trust it.
If asked to design a grader on a whiteboard, name the options before you pick. Each has a distinct failure mode and the strongest answers fuse two or three rather than betting on one.
Learn more
Advanced table
The strongest answers fuse two or three rather than betting on one
- Grader type
- Execution / partial-credit
- Reward it produces
- Fraction of tests passing, build success, lint clean, runtime checks
- Where it breaks
- Needs an executable task; rewards test-editing; blind to quality and intent
- Grader type
- LLM-as-judge + rubric
- Reward it produces
- Score from a model reading the diff against written criteria
- Where it breaks
- Verbose-bias, position-bias, inconsistent; the judge can be fooled or hacked
- Grader type
- Learned reward model
- Reward it produces
- Scalar from a model trained on human preference pairs (RLHF-style)
- Where it breaks
- Only as good as the labels; drifts off-distribution; expensive to keep fresh
- Grader type
- Hybrid / gated
- Reward it produces
- Execution gate first, then a judge or RM scores the survivors
- Where it breaks
- More moving parts; you must trust each stage's contribution
| Grader type | Reward it produces | Where it breaks |
|---|---|---|
| Execution / partial-credit | Fraction of tests passing, build success, lint clean, runtime checks | Needs an executable task; rewards test-editing; blind to quality and intent |
| LLM-as-judge + rubric | Score from a model reading the diff against written criteria | Verbose-bias, position-bias, inconsistent; the judge can be fooled or hacked |
| Learned reward model | Scalar from a model trained on human preference pairs (RLHF-style) | Only as good as the labels; drifts off-distribution; expensive to keep fresh |
| Hybrid / gated | Execution gate first, then a judge or RM scores the survivors | More moving parts; you must trust each stage's contribution |
Four grader types, each trustworthy in a different regime and each with its own failure mode - the strongest answers fuse two or three rather than betting on one.