2 min lesson
Keep the path lean
Name the key items in "Keep the path lean", then explain why each one matters.
Step 1 of 3
Keep the path lean
The gateway should add as little of its own latency as possible. The dominant cost is the provider's generation time, so the gateway's overhead has to be a rounding error, not a tax.
- Pool connections and keep them warm so no hot request pays for a handshake.
- Use async, non-blocking I/O end to end; one slow upstream must never starve a worker that other requests need.
- Never block the event loop with synchronous work - parsing, hashing or accounting on the critical pathThe longest end-to-end chain of dependent work in a plan; it sets the realistic timeline no matter how parallel everything else looks. A slip on the critical path moves the date; a slip on a task with slack does not. Press Enter for the full definition. belongs off it.
- Stream straight through; buffering a response adds latency equal to the entire generation time.
Learn more
Full explanation
Hedging is a measured threshold, not a vibe
Hedging is a measured threshold, not a vibe
Hedging is the strongest lever against a slow provider tail and it's only worth its cost if you fire it at the right moment. Set the trigger from the live latency distribution.
Setting the hedge threshold
- Anchor on p95
- Fire the hedge when a request crosses the measured p95, so you only duplicate the slow ~5% - not every request.
- Recompute continuously
- The distribution drifts with provider health and time of day; a static threshold is wrong by lunch.
- Bound the extra spend
- A 5% hedge rate is ~5% extra cost on those requests - acceptable for Tab, often not for a long Agent turn.
- Cancel aggressively
- Kill the loser on the winner's first token so you pay for a sliver of overlap, not two full generations.