Routing, Failover & Backpressure: The Core Systems Round

Every Tab completion, every Agent step, every chat turn in Cursor exits through one door. The inference gateway is that door and the JD describes the job as building a single abstraction over every provider's API so onboarding a new model is a config change, not a code change.

Anthropic, OpenAI, Google and a fleet of self-hosted models each speak a different dialect. Their auth differs, their request and response shapes differ, their streaming framing differs and their error vocabularies disagree about what a 429 even means. The gateway's first job is to swallow that heterogeneity so the rest of the system reasons about one model interface.

Frame it to an interviewer as an adapter pattern with teeth. A thin per-provider adapter translates the wire format. A fat shared core owns everything that should never be reimplemented per provider: timeouts, retries, tracing, token accounting, idempotency.

What the gateway normalizesthe dialect problem

Name the specific seams that diverge across providers. Vague "it abstracts the API" reads as someone who has never onboarded a second provider. The texture below is what reads as having shipped one.

The split: thin adapter, fat core

Decide deliberately what lives in the adapter versus the shared core. Put too much in the adapter and every provider reimplements retries badly. Put provider quirks in the core and you leak the abstraction.

Streaming passthrough is first-class

The single biggest gateway mistake is buffering the full response before forwarding it. That destroys time-to-first-token, which is the latency a developer actually feels when text starts appearing in the editor.

• Forward token deltas as they arrive; never wait for the final chunk to start the client stream.
• Keep the gateway's per-request memory flat - proxy the stream, don't accumulate it.
• Tap the stream for metrics (TTFT, inter-token latency, token counts) without blocking forwarding.
• Propagate client cancellation downstream so an abandoned Agent turn stops burning provider tokens.

Adding a model is a config change

This is the line from the JD, so make it concrete. New models that share an existing provider's API should require zero code: a declarative entry naming the provider, model id, limits and pricing.

models:
  claude-sonnet-tab:
    provider: anthropic        # reuses the existing adapter
    upstream_id: claude-sonnet-4.6
    surfaces: [tab]
    limits: { rpm: 4000, tpm: 2_000_000 }
    timeout_ms: 800            # tight: Tab is sub-100ms-budget critical
    price_per_mtok: { in: 3.00, out: 15.00 }
    fallback: claude-haiku-tab

The JD sets a hard bar: no single provider outage causes user-visible degradation. When Anthropic browns out, Tab and Agent should keep working and most users should never notice.

Failover is layered defense, not one trick. Health-based routing keeps traffic off a sick provider before a request is even sent. Circuit breakers stop you from hammering a dying upstream. Bounded retries with jitter recover transient blips. A fallback ladder catches the rest. Each layer covers a failure the others miss.

The resilience toolkitand what each one is actually for

Retries with a budget, not a count

The naive answer is "retry three times." The senior answer is a retry budget: cap retries as a fraction of total traffic so that when a provider fails broadly, retries can't multiply load and turn a partial outage into a total one.

1. 1Classify the failure. Only retry what the normalized taxonomy marks retryable - a 503 or a timeout, never a 400 or a content-policy refusal.
2. 2Back off with jitter. Exponential backoff plus randomization spreads the retry wave so a thousand clients don't all retry on the same tick.
3. 3Spend from a global budget. If retries exceed, say, 10% of requests, stop retrying and shed - the provider is broadly down, not blipping.
4. 4Cross over, don't just repeat. A retry that targets the same dead provider is wasted; route the retry to the next provider on the ladder.

Hedging: buy tail latency with cost

Hedging fires a second request to a different provider after a delay, then takes whichever responds first and cancels the loser. It directly attacks the slow tail and it directly costs duplicate spend.

Idempotency under retry and hedge

The moment you retry or hedge, the same logical request can hit upstreams twice. Without idempotency you double-charge the user, double-count tokens or double-apply a side effect.

• Stamp every logical request with an idempotency key at admission, before any retry or hedge can fork it.
• Dedup at the gateway: the first response to arrive wins, later duplicates are dropped, not forwarded.
• Make completions side-effect-free at the gateway layer so a duplicate is wasteful but never incorrect.
• Reconcile token usage against the winning response only, so cost accounting matches what the user actually received.

Quality-aware fallback

A fallback model is usually cheaper or faster and lower quality. That tradeoff is acceptable for Tab, where a slightly weaker completion beats no completion and far less acceptable for an Agent doing a multi-file refactor. Decide per surface and make the decision observable.

The JD frames this as making sure traffic spikes don't cascade into providers. Backpressure is the system saying no early and cleanly, so a surge doesn't melt the fleet or trip every provider's rate limiter at once.

The instinct under load is to queue everything. That's the trap. An unbounded queue trades a fast failure for a slow one: requests sit until they're stale, the client times out and retries and you've added load to a system that's already over capacity. Admission control decides what to accept before any of that happens.

The control surfacesfrom the edge inward

Shape traffic before the provider rejects it

Providers publish RPM and TPM limits. If you let traffic blow past them, the provider returns 429s, your retry logic kicks in and you've manufactured a retry storm out of your own success. The fix is to enforce the provider's budget proactively at the gateway.

1. 1Model the provider's budget locally. Mirror each provider's RPM/TPM as a token bucket the gateway owns, refilled at the published rate.
2. 2Admit against the bucket. A request that would exceed the budget is shed or queued briefly at the gateway - never forwarded to be rejected upstream.
3. 3Reserve headroom. Run the bucket below the hard limit so a burst doesn't tip you over and so failover traffic from another provider has room to land.
4. 4Adapt to reality. If the provider starts 429-ing despite your model, treat that as a signal to throttle harder, not to retry into it.

Priority and QoS classes

Not all traffic is equal. A Tab completion has a sub-100ms budget and a user waiting on a keystroke. A background re-index or a bulk eval job can wait. Under contention, the latency-critical class must win.

Cursor's request path is always hot and always user-facing. Optimize for the tail, because the p99 is what a developer feels - and for Tab, the whole budget is under 100ms, so the tail is the product.

The mean lies. If p50 is 40ms and p99 is 600ms, one in a hundred completions arrives after the user has typed three more characters and they experience that as broken even though the average looks great. Talk in percentiles or an inference interviewer will assume you've never operated a serving path.

Where the tail comes fromname it to mitigate it

Keep the path lean

The gateway should add as little of its own latency as possible. The dominant cost is the provider's generation time, so the gateway's overhead has to be a rounding error, not a tax.

• Pool connections and keep them warm so no hot request pays for a handshake.
• Use async, non-blocking I/O end to end; one slow upstream must never starve a worker that other requests need.
• Never block the event loop with synchronous work - parsing, hashing or accounting on the critical path belongs off it.
• Stream straight through; buffering a response adds latency equal to the entire generation time.

Hedging is a measured threshold, not a vibe

Hedging is the strongest lever against a slow provider tail and it's only worth its cost if you fire it at the right moment. Set the trigger from the live latency distribution.

Budget the milliseconds end to end

Know where the time goes: client to gateway, admission and routing, gateway to provider, provider generation, stream back. For a sub-100ms Tab path, every hop has a number and you should be able to name which hop you'd cut first.

The JD asks for on-call-grade reliability on a path that's always hot. That means you operate it like an owner: explicit SLOs, an error budget, instrumentation that localizes a problem in minutes and rollouts that can't take everyone down at once.

Senior IC from week one means you don't wait for someone else to define "good." You name the SLOs, you wire the metrics and you build the kill switch before you ship the model that needs it.

SLOs you'd actually commit toper surface, not one global number

Metrics: golden signals plus inference-specific

The four golden signals (latency, traffic, errors, saturation) are table stakes. The inference-specific metrics are what tell an interviewer you know this domain.

Tracing across providers

When a request is slow, you need to know which hop in seconds, not after an hour of log-grepping. A trace that spans admission, routing, the gateway adapter and the provider call localizes latency and failure immediately.

• Propagate a trace and idempotency id from admission through every retry, hedge and fallback so duplicated work is still one logical trace.
• Span each hop separately - routing decision, adapter serialize, provider TTFT, stream duration - so the slow hop is obvious.
• Tag spans with provider, model, surface and outcome (served / failed-over / shed) so you can slice the tail by any dimension.

Safe rollout of a new model or provider

A new model on the hot path can degrade quality or latency for everyone instantly. Roll it out so a mistake is contained and reversible in seconds.

1. 1Shadow first. Mirror real traffic to the new model without serving its output; compare latency, cost and quality offline with zero user risk.
2. 2Canary small. Route a tiny slice of live traffic, watch the per-surface SLOs and failover rate and hold before widening.
3. 3Keep a kill switch. A config flag flips the model out of rotation instantly - rollback is a config push, not a deploy.
4. 4Ramp on green. Widen the canary only while the error budget is healthy; freeze the ramp the moment it isn't.

The strongest onsite design answer is a single request narrated end to end. Trace one completion from the keystroke to the streamed token and show where every technique from this module plugs in.

This is the backbone of the systems-design round. Walk the happy path once, then walk the failure branches, naming the mechanism at each step rather than gesturing at "the system handles it."

The happy path, step by stepkeystroke to token

1. 1Admission. The request arrives tagged with its surface and an idempotency key. Token bucket and concurrency check: under budget, admit; over budget, shed fast with a retryable error.
2. 2Routing decision. Pick model and provider from config and live health, with cache affinity - route to where a matching prefix is already cached if there is one.
3. 3Gateway adapter. Serialize the canonical request into the chosen provider's dialect, attach a tight timeout for the surface and open the upstream stream.
4. 4Provider + stream. Tokens stream back through the gateway untouched - forwarded delta by delta to preserve TTFT, tapped for metrics, never buffered.
5. 5Metrics + accounting. On completion, record TTFT, ITL, token usage and cost against the idempotency key; close the span.

Where each technique plugs in

Tab versus Agent: same path, different dials

One pipeline serves both, but the parameters invert. Contrasting them shows you understand that the design is a set of dials, not a single fixed policy.

The failure branches

The happy path is the easy half. Naming the branches - and the clean exit when there's no good option left - is what closes a strong design answer.

• Provider returns 5xx or times out → classify as retryable, spend from the retry budget, cross-route to the next provider on the ladder.
• Over admission budget → shed fast with an explicitly-retryable error and a backoff hint, never queue past the latency budget.
• Oversized context → down-route to a larger-window model or reject with a clear, actionable error before wasting a provider call.
• Last rung of the ladder is unhealthy → degrade gracefully per surface and if nothing is viable, return a clean retryable error rather than silently worse output.