1 min lesson
Metrics: golden signals plus inference-specific
Use "The inference-specific metrics are what tell an interviewer you know this domain" to describe the practical rule.
Step 1 of 3
Metrics: golden signals plus inference-specific
The four golden signals (latency, traffic, errors, saturation) are table stakes. The inference-specific metrics are what tell an interviewer you know this domain.
Learn more
Advanced table
Failover rate and shed rate are the two metrics most people forget
- Metric
- TTFT
- What it tells you
- Time to first token - the latency users feel
- Alert when
- p99 breaches the surface's budget
- Metric
- Inter-token latency (ITL)
- What it tells you
- Stream smoothness during generation
- Alert when
- ITL spikes - provider is degrading mid-stream
- Metric
- Cache hit rate
- What it tells you
- Prefix/response cache effectiveness
- Alert when
- Drops sharply - a cost and latency regression
- Metric
- Failover rate
- What it tells you
- How often you leave the primary provider
- Alert when
- Rises - primary is unhealthy or quality-degrading
- Metric
- Shed rate
- What it tells you
- How much load you're rejecting
- Alert when
- Non-zero and sustained - you're under-provisioned
- Metric
- Per-provider error rate
- What it tells you
- Which upstream is sick
- Alert when
- One provider diverges from the pack
| Metric | What it tells you | Alert when |
|---|---|---|
| TTFT | Time to first token - the latency users feel | p99 breaches the surface's budget |
| Inter-token latency (ITL) | Stream smoothness during generation | ITL spikes - provider is degrading mid-stream |
| Cache hit rate | Prefix/response cache effectiveness | Drops sharply - a cost and latency regression |
| Failover rate | How often you leave the primary provider | Rises - primary is unhealthy or quality-degrading |
| Shed rate | How much load you're rejecting | Non-zero and sustained - you're under-provisioned |
| Per-provider error rate | Which upstream is sick | One provider diverges from the pack |
Failover rate and shed rate are the two metrics most people forget - they're where blast radius shows up first.
"I'd commit to per-surface SLOs, not one global number. Tab availability is the strictest budget - a failed Tab flickers on every keystroke - and Tab p99 has to sit inside the sub-100ms end-to-end budget; Agent tolerates a retryable hiccup mid-task, and cost-per-request gets its own budget per surface. I manage all of that to an error budget: when it's burned, risky rollouts freeze until it recovers. For blast radiusHow much breaks if a change goes wrong; the scope of potential damage. Press Enter for the full definition., the two metrics I watch hardest are failover rate and shed rate - a rising failover rate says I'm leaving the primary provider at scale, a sustained shed rate says I'm rejecting load, and both move before raw availability craters, so they're where a spreading problem surfaces first."