1 min lesson
Cover the cases that bite, not just the happy path
Rebuild the main list in "Cover the cases that bite, not just the happy path", then say what each item changes.
Step 1 of 2
Cover the cases that bite, not just the happy path
A demo passes on the file you picked because it looks clean. Production fails on the file with a circular import, the generated code, the 2,000-line module nobody wants to touch. Seed your eval set with the ugly ones on purpose.
- Adversarial inputs: malformed files, partial edits, ambiguous instructions, files that shouldn't be touched at all.
- Edge cases from the real repo: generated code, vendored deps, the one file with no test coverage.
- Negative tests: the agent should sometimes refuse or escalate and the eval should reward that.
Watch out
An eval set that only contains tasks the agent already passes is theater. It produces a comforting 100% and predicts nothing. Deliberately include tasks you expect to fail today - the gap is your roadmap and watching it close is the proof of progress.