1 min lesson
Measuring a non-deterministic agent
Answer "Why can't you trust a single proxy signal like 'edit undone' as a quality label?" Use one lesson detail to support it.
Step 1 of 3
You cannot grade a single agent run the way you grade a unit test. The same prompt can produce a brilliant edit once and a wrong one the next. Measure the population, not the run.
Cursor's product is non-deterministic by design, so the metric has to be statistical from the start. Define success as a rate over many runs - share of runs that reach an accepted, correct edit - and watch its distribution and variance, not a single trace.