Skip to lesson
Exit
Coding & Engineering Craft: TS, Rust, Python on the Hot Path1 / 2

1 min lesson

Hedging without doubling the bill

Respond to "Why is request hedging a double-edged tool and how do you keep it from blowing up cost?" Name the reason and the detail behind it.

Step 1 of 2

Hedging without doubling the billlatency vs cost, made explicit

  1. 1Fire the primary. Start the request and a timer for the hedge delay (often near the p95 of normal latency).
  2. 2Hedge only on the tail. If the primary hasn't returned by the delay, launch a second attempt - ideally to a different provider.
  3. 3Take the first, cancel the rest. Resolve on the first success and abort the loser so it stops consuming tokens and GPU.
  4. 4Cap the duplicate cost. Hedge a bounded fraction of traffic; if hedge rate climbs, that's a signal the primary is unhealthy, not a reason to hedge everything.
Idempotency makes retries safe

Retries and hedges both mean a request can execute twice. An idempotency key per logical request lets the server dedupe, so two attempts that both land don't double-apply a side effect. For pure read-like inference calls this is cheap; the moment a request writes (billing, a stored completion) it is mandatory.

Watch out

Cancellation has to propagate all the way down. If the client disconnects but you only resolve the top-level promise, the upstream provider call keeps running and keeps billing. Thread one AbortSignal from the edge through every await and demonstrate that in the code.