1 min lesson
Surface failures to a human who can act
Rebuild the main list in "Surface failures to a human who can act", then say what each item changes.
Step 1 of 2
Surface failures to a human who can act
A metric that drops is a notification. A useful notification carries the trace link, the failing input, the grader's reason and a guess at the category. The on-call engineer should be able to click once and see exactly what the agent saw.
- Link every alert to the specific trace and the prompt/context that produced it, so root cause is one click away.
- Tag failures by category at capture time (bad context, wrong tool, timeout) so patterns aggregate instead of scattering.
- Keep enough context to reproduce without the customer's help - assume they won't or can't re-run it for you.
Interview move
In the system-design round, when asked to design an AI workflow, draw the observability layer unprompted. Say: every call emits a trace, traces roll up into success rate / p95 / cost-per-task and the eval set runs on a schedule against the same traces to catch drift. That one paragraph signals you've operated these systems, not just built them.