2 min lesson
What to capture on every model call
Use "What to capture on every model call" to explain each part and the role it plays.
Step 1 of 3
What to capture on every model call
Per-call trace fields
- Inputs + context
- The exact prompt, retrieved files and rules - the agent's full view of the world
- Output
- Raw model response before any post-processing
- Tool calls
- Which tools fired, with what arguments and what they returned
- Tokens
- Prompt and completion counts, the input to cost and the signal for context bloat
- Latency
- Wall-clock per call and per task, broken out by step
- Cost
- Dollars per task, derived from tokens and model
- Outcome
- The grader verdict for this run, linked back to the trace
If a field isn't captured at runtime, it doesn't exist when you need to debug.
Learn more
Advanced table
The metrics that actually tell you something
The metrics that actually tell you somethingaggregate the traces
- Metric
- Success rate
- What it answers
- Is the workflow still doing its job?
- When it should alarm
- Drops below the gate threshold over a rolling window
- Metric
- p50 / p95 latency
- What it answers
- Is it fast enough to use, including the slow tail?
- When it should alarm
- p95 crosses the point where users abandon it
- Metric
- Cost per task
- What it answers
- Is the unit economics still sane?
- When it should alarm
- Creeps up - usually context bloat or extra retries
- Metric
- Error / retry rate
- What it answers
- Is the plumbing healthy underneath the model?
- When it should alarm
- Spikes from timeouts, rate limits or tool failures
| Metric | What it answers | When it should alarm |
|---|---|---|
| Success rate | Is the workflow still doing its job? | Drops below the gate threshold over a rolling window |
| p50 / p95 latency | Is it fast enough to use, including the slow tail? | p95 crosses the point where users abandon it |
| Cost per task | Is the unit economics still sane? | Creeps up - usually context bloat or extra retries |
| Error / retry rate | Is the plumbing healthy underneath the model? | Spikes from timeouts, rate limits or tool failures |
Track p95, not just the average. The average hides the runs that make people give up.
Drift is the silent killer
The scary failure isn't a crash, it's a slow slide. A provider ships a new model snapshot, your context grows as the repo grows and success rate erodes a point a week. Nothing errors. The way you catch it is by running the eval set on a schedule, not just at build time and alerting when the rolling success rate or cost per task moves off its baseline.
Learn more
Optional practice
Practice: What to capture on every model call
QWhy is tracking p95 latency more important than tracking average latency for a model-backed workflow?