Skip to lesson
Exit
Evals & Production Reliability1 / 2

1 min lesson

Designing an eval set for code-modifying AI

Walk through "Designing an eval set for code-modifying AI" in order, then name the proof that tells you it worked.

Step 1 of 2

An eval set is a list of tasks where you already know the right answer, plus a grader that can decide automatically whether the agent got there. For code, you're lucky: the compiler and the test suite are graders that don't get tired.

Start by collecting golden tasks - concrete inputs with a known-good outcome. For a refactor workflow a golden task might be a real file and the assertion "after the edit this still compiles, the existing tests pass and the public behavior is unchanged." You don't need a thousand of these. You need fifteen that cover the cases you actually care about.

  1. 1Pull real examples. Mine the customer's repo and git history for representative tasks, not toy snippets. Adversarial cases live in their codebase already.
  2. 2Pin the known-good outcome. Write down what “correct” means per task: compiles, tests pass, type-checks, behavior preserved, no new lint errors.
  3. 3Pick a grader per task. Choose the cheapest grader that actually catches the failure you care about.
  4. 4Set a gate. Decide the pass rate below which a rollout is blocked and stick to it under deadline pressure.
  5. 5Run it on every change. If it's slow or expensive, you'll stop running it and an eval you don't run is a comment.
Learn more

Advanced table

Choosing graders - and knowing how each one lies

Choosing graders - and knowing how each one liesevery grader has a failure mode

Grader
Exact match
Good for
Deterministic output, format checks, small string transforms
How it fails you
Brittle - a valid but differently-formatted answer fails; useless for open-ended edits
Grader
Test-pass
Good for
Behavior preservation on refactors and migrations
How it fails you
Only as good as coverage; green tests on untested code prove little
Grader
Type-check / compile
Good for
Catching structurally broken edits cheaply and fast
How it fails you
Compiles ≠ correct; logic bugs sail straight through
Grader
LLM-as-judge
Good for
Subjective quality: is this explanation clear, is this the idiomatic fix
How it fails you
Non-deterministic, can be gamed, drifts with the judge model; needs its own calibration

Layer graders: cheap deterministic checks first, an LLM judge only for what they can't see.

THE LAYERED GRADER PIPELINE

Interactive diagram. Step through it with the Next and Previous controls below, or Tab to a region to read its detail.

diagram: flow

Cheap, deterministic checks first; the model call last; the gate decides rollout.

A golden task + a layered grader, sketched in Python
golden = {
    "id": "refactor-auth-001",
    "input_file": "src/auth/session.py",
    "instruction": "Replace the deprecated session API with the v2 client.",
    "expect": {"compiles": True, "tests_pass": True, "behavior": "preserved"},
}

def grade(task, agent_output):
    if not type_checks(agent_output):        # cheap, deterministic
        return Fail("type_error")
    if not run_tests(task.tests):            # behavior preservation
        return Fail("test_regression")
    verdict = llm_judge(task, agent_output)  # only for taste
    return verdict  # Pass / Fail + reason
Learn more

Optional practice

Practice: Designing an eval set for code-modifying AI

QWhy layer a compile check and a test-pass check before an LLM-as-judge grader, instead of just using the judge?