1 min lesson
Stretch: hedged requests with a cost cap
Show why this lesson detail matters: "Hedged requests with a cost cap."
Step 1 of 2
Stretch: hedged requests with a cost cap
If you finished early, add request hedging: when TTFT exceeds the p95 budget, fire a second request to a backup provider and take whichever streams first, then cancel the loser. The cap is the senior detail - at most one hedge per request, because a naive hedge doubles your provider bill at exactly the moment the fleet is already struggling.
Cursor lets you use AI in the screen and then demands you understand every line. After your solve, close the editor and re-implement the timeout-and-cancel logic from memory on paper. If you can't, a raw-output submission would have failed the round - the interviewer's follow-up (“why full jitter and not just exponential?”) is exactly the question raw paste can't survive.
Learn more
Optional practice
Practice: Stretch: hedged requests with a cost cap
QWhy is it unsafe to retry a streaming inference request after the first token has already been sent to the user?