1 min lesson
Non-determinism is the constraint that shapes every metric you design
Give a plain answer to "Why must reliability metrics for an agent product be statistical and population-level rather than per-trace?" Then ground it in one lesson detail.
Step 1 of 2
Non-determinism is the constraint that shapes every metric you design. The same prompt run twice can take different paths, call different tools and finish in different times. So a single trace proves almost nothing. Metrics have to be statistical and population-level: rates and percentiles over many runs, not the story of one run.
- p50 hides pain
- The median user is usually fine; the misery lives further out.
- p95 / p99 reveal it
- Tail latency and rare failures are what 'disproportionate frustration' means.
- Billions amplify rare
- A 0.1% failure mode is millions of broken interactions per period.
Don't blame the model by reflex. In the loop, when you're handed a regression, the strong move is to ask whether the harness changed - a deploy, a tool, a timeout config - before assuming the LLM got worse. Most reliability regressions are harness regressions and a candidate who jumps straight to 'the model degraded' reads as someone who hasn't run an agent product.