1 min lesson
Hedging: buy tail latency with cost
Walk through each part of "Hedging: buy tail latency with cost", then explain what each one does.
Step 1 of 3
Hedging: buy tail latency with cost
Hedging fires a second request to a different provider after a delay, then takes whichever responds first and cancels the loser. It directly attacks the slow tail and it directly costs duplicate spend.
How to hedge without lighting money on fire
- Pick by surface
- Hedge Tab aggressively (cheap, latency-critical); hedge a long Agent turn rarely (a duplicate is expensive).
- Respect idempotency
- Two in-flight copies must dedup so you don't apply or bill the same effect twice.
Learn more
Full explanation
Idempotency under retry and hedge
Idempotency under retry and hedge
The moment you retry or hedge, the same logical request can hit upstreams twice. Without idempotency you double-charge the user, double-count tokens or double-apply a side effect.
- Stamp every logical request with an idempotency key at admission, before any retry or hedge can fork it.
- Dedup at the gateway: the first response to arrive wins, later duplicates are dropped, not forwarded.
- Make completions side-effect-free at the gateway layer so a duplicate is wasteful but never incorrect.
- Reconcile token usage against the winning response only, so cost accounting matches what the user actually received.