1 min lesson
Offline eval vs. online telemetry
Rebuild the parts of "Offline eval vs. online telemetry", then say why each one matters.
Step 1 of 2
Offline eval vs. online telemetrytwo truths that sometimes disagree
When to trust which
- Offline eval
- controlled, repeatable, labeled - trust for pre-ship gating and regression catches
- Online telemetry
- real users, real repos, real distribution - trust for what actually happens in production
- When they disagree
- usually your eval set is unrepresentative; treat the gap as a signal to fix the eval, not to dismiss telemetry
Interview move
When asked how you'd measure agent quality, refuse the single-run framing out loud: "You can't grade one run - it's stochastic, so I'd define a population success rate and triangulate with acceptance, undo and re-prompt as proxies." Naming the stochastic nature first is the senior signal in this round.