Skip to lesson
Exit
Routing, Failover & Backpressure: The Core Systems Round1 / 3

1 min lesson

Tracing across providers

Recall the main items in "Tracing across providers", then connect each one to the work.

Step 1 of 3

Tracing across providers

When a request is slow, you need to know which hop in seconds, not after an hour of log-grepping. A trace that spans admission, routing, the gateway adapter and the provider call localizes latency and failure immediately.

  • Propagate a trace and idempotency id from admission through every retry, hedge and fallback so duplicated work is still one logical trace.
  • Span each hop separately - routing decision, adapter serialize, provider TTFT, stream duration - so the slow hop is obvious.
  • Tag spans with provider, model, surface and outcome (served / failed-over / shed) so you can slice the tail by any dimension.
Learn more

Full explanation

Safe rollout of a new model or provider

Safe rollout of a new model or provider

A new model on the hot path can degrade quality or latency for everyone instantly. Roll it out so a mistake is contained and reversible in seconds.

  1. 1Shadow first. Mirror real traffic to the new model without serving its output; compare latency, cost and quality offline with zero user risk.
  2. 2Canary small. Route a tiny slice of live traffic, watch the per-surface SLOs and failover rate and hold before widening.
  3. 3Keep a kill switch. A config flag flips the model out of rotation instantly - rollback is a config push, not a deploy.
  4. 4Ramp on green. Widen the canary only while the error budget is healthy; freeze the ramp the moment it isn't.
Blast-radius thinking

Multi-region and multi-provider routing isn't just for failover - it caps how much a single bad change can break. A canary scoped to one region, one surface and one provider means a regression hits a sliver of traffic and a config-driven kill switch pulls it back before the error budget notices. Mention data-locality too: some traffic may be pinned to a region for compliance and your routing has to honor that constraint, not optimize around it.

Learn more

Optional practice

Practice: Tracing across providers

QBeyond the four golden signals, which two inference-specific metrics most directly reveal that your blast radiusHow much breaks if a change goes wrong; the scope of potential damage. Press Enter for the full definition. is growing and why?