1 min lesson
Whiteboard: GPU capacity plan
Put this idea into your own words: "The prompt: a new feature is projected to double request volume in eight weeks - plan the GPU capacity."
Step 1 of 4
Whiteboard, 45 minutes, one senior leader watching. The prompt: a new feature is projected to double request volume in eight weeks - plan the GPU capacity. This is the systems-literacy round wearing a planning hat. They're checking whether you can reason about utilization, latency and the cost of guessing wrong in both directions.
Don't draw boxes first. Quantify the demand, then split it into a baseline you reserve and a peak you burst, then name the trigger that moves capacity. Every number you say buys you the right to the next design choice.
Learn more
Full explanation
Whiteboard: GPU capacity plan
- 1Demand model. Tie GPU need to usage, not vibes: requests/sec at peak = DAU × calls-per-user-day × peak-concentration ÷ seconds. Doubling request volume doesn't double GPUs if utilization has headroom - say that. Translate requests/sec into GPUs via throughput per GPU at your latency SLA.
- 2Reserve vs. burst split. Reserve the steady baseline where committed pricing wins, burst the spiky top on on-demand. The breakeven is sustained utilization: reserved capacity pays off above roughly 60–80% sustained use, so reserve only what you'll actually keep busy and let bursty demand ride on-demand even at a higher unit price.
- 3Utilization target. Name a target band and defend it. GPU fleets commonly run 15–30% utilized, which is money on the floor; you want the reserved tier pushed higher without starving latency headroom. State the latency SLA first - tab prediction needs tens-of-milliseconds responses - because that ceiling caps how hard you can pack a GPU before the tail blows out.
- 4Trigger points. Pre-commit the rules: scale up when sustained utilization crosses your high-water mark for N minutes, add reserved capacity when the burst bill exceeds reserved cost for a sustained stretch and set a hard latency-SLA breach alarm that scales out regardless of cost.
Learn more
Advanced table
Name the cost of being wrong both ways
Name the cost of being wrong both waysThe clause that separates a planner from a forecaster
- Direction
- Over-provision
- What it costs
- Idle reserved GPUs burn committed spend at 15–30% utilization - the default failure of capacity teams
- How the plan hedges
- Reserve only the defensible baseline; let the uncertain top ride burst
- Direction
- Under-provision
- What it costs
- Latency SLA breaches, queueing, degraded tab/agent UX at launch - a product hit, not just a metric
- How the plan hedges
- Burst headroom + a latency-breach trigger that scales out regardless of cost
- Direction
- Wrong tier mix
- What it costs
- Large models serving simple calls inflates $/request even at good utilization
- How the plan hedges
- Model routing so small models take the simple share before you buy more GPUs
| Direction | What it costs | How the plan hedges |
|---|---|---|
| Over-provision | Idle reserved GPUs burn committed spend at 15–30% utilization - the default failure of capacity teams | Reserve only the defensible baseline; let the uncertain top ride burst |
| Under-provision | Latency SLA breaches, queueing, degraded tab/agent UX at launch - a product hit, not just a metric | Burst headroom + a latency-breach trigger that scales out regardless of cost |
| Wrong tier mix | Large models serving simple calls inflates $/request even at good utilization | Model routing so small models take the simple share before you buy more GPUs |
Saying both failure costs out loud is the judgment signal; picking a hedge for each is the plan.
“Doubling requests doesn't mean doubling GPUs if we're sitting at 25% utilization - first I'd model whether routing simple calls to a small model absorbs the increase. I'd reserve the baseline we keep busy above ~70% and burst the launch spike on-demand, because over-reserving leaves idle GPUs burning committed spend and under-reserving breaks the tab-latency SLA at launch. My trigger to add reserved capacity is the burst bill exceeding reserved cost for two sustained weeks.”
When the senior leader pushes back mid-answer - “why 70%, that seems low” - don't defend the number, defend the method. Say what 70% is protecting (latency headroom for the spiky tail) and what would move it (“if the workload were batchable and latency-loose, I'd pack to 85%”). Updating your number on a better fact while holding the reasoning is exactly the spirited-debate, truth-seeking behavior the round is built to test. Caving instantly reads as no conviction; refusing to move reads as no curiosity.
Watch the clock and finish. A complete, defensible plan with all four parts beats an elegant demand model you never connected to a reserve-vs-burst decision. If you're at minute 35 with no trigger points, stop refining the forecast and state the triggers - interviewers grade the whole framework and an unfinished plan signals you'd run a real program the same way.
QA feature is projected to double request volume. Why might you not need to double GPU capacity and what is the first thing you'd check?