1 min lesson
Variance and significance
Use the example in "Variance and significance" to explain the main idea in plain words.
Step 1 of 2
Variance and significance
Agent evals are noisy. Sampling temperature, tool-call ordering and flaky environments mean two runs of the same model disagree. A single run that's 2 points higher tells you nothing.
Don't report a point estimate from one seed - report a distribution
# Bad: one run, no error bars score = run_eval(model) # 47.3% - looks like a win vs 45.1%? # Better: many seeds/tasks, with a confidence interval scores = [run_eval(model, seed=s) for s in range(N_SEEDS)] mean, ci = bootstrap_ci(scores) # 46.1% ± 2.4% -> the "win" is in the noise # Decide with a test, not a vibe significant = paired_test(baseline_scores, new_scores).p < 0.05
Watch out
Enough tasks and seeds to make the confidence interval smaller than the effect you care about and a paired test against the baseline on the same tasks. If your interval is ±2.4 and your gain is +2.2, you have not measured an improvement. Reporting a single-run delta as a result is the fastest way to lose a technical interviewer's trust.