1 min lesson
Eval design for coding agents
Put this idea into your own words: "An eval you can't trust is worse than no eval, because it makes you confidently wrong."
Step 1 of 3
An eval you can't trust is worse than no eval, because it makes you confidently wrong. The whole game is building measurements whose movement you'd bet a model release on.
Start by knowing the landscape so you can speak to it. Public benchmarks tell the interviewer you read the field; internal evals tell them you understand why public ones aren't enough.
Learn more
Advanced table
Public benchmarks are a sanity floor, not the target
- Eval
- SWE-bench / SWE-bench Verified
- What it measures
- End-to-end fix of real GitHub issues, scored by hidden tests
- Limit
- Public, so contaminated; one distribution; saturating
- Eval
- HumanEval / MBPP
- What it measures
- Function-level code generation from a docstring
- Limit
- Tiny, fully leaked, not agent-assisted, far from real work
- Eval
- Internal task evals
- What it measures
- Agent on tasks drawn from real developer requests
- Limit
- Private and fresh, but expensive to build and label
- Eval
- Online product metrics
- What it measures
- Acceptance rate, edit-retention, follow-up rate in the live product
- Limit
- The truth, but slow, confounded and lagging
| Eval | What it measures | Limit |
|---|---|---|
| SWE-bench / SWE-bench Verified | End-to-end fix of real GitHub issues, scored by hidden tests | Public, so contaminated; one distribution; saturating |
| HumanEval / MBPP | Function-level code generation from a docstring | Tiny, fully leaked, not agent-assisted, far from real work |
| Internal task evals | Agent on tasks drawn from real developer requests | Private and fresh, but expensive to build and label |
| Online product metrics | Acceptance rate, edit-retention, follow-up rate in the live product | The truth, but slow, confounded and lagging |
Public benchmarks are a sanity floor, not the target. The evals that decide a Cursor release are private, distribution-matched and tied to shipped behavior.
Learn more
Optional practice