2 min lesson
The levers that raise it
Respond to "Why is a high failure-and-retry rate worse for cost than its reliability number suggests?" Name the reason and the detail behind it.
Step 1 of 2
The levers that raise itstandard moves, in order of usual payoff
Fit workloads onto fewer GPUs
Reduces stranded fractions of cards
Attacks allocation-vs-compute gap directly
Match capacity to demand over time
Shed idle hours in troughs
Bounded by GPU acquisition lead time
Run latency-tolerant batch on the same fleet
Backfills the gaps left by spiky inference
Needs isolation so it does not hurt the hot path
Failed work burns GPU time with zero output
Retry storms quietly destroy utilization
Track failure rate as a cost metric, not just reliability
That last card surprises people. A 10% failure-and-retry rate is not only a reliability problem - it means a tenth of your GPU-seconds produced nothing, then got spent again on the retry. Failed work is doubly expensive and it hides inside utilization that looks busy.
- Target band
- Set an explicit range (for example 60-80% compute utilization) per workload class
- Floor
- Below it, capacity is over-provisioned - shrink the reservation or co-locate
- Ceiling
- Above it, you have no headroom for burst - the next launch will be capacity-starved
- Cadence
- The recurring review enforces the band; a metric with no forum is a metric nobody manages
A band, not a single number - too low wastes money, too high removes safety margin.
Our fleet is 95% allocated but 28% busy, so the prize is the gap, not more cards. I would set a 65% compute-utilization target band per workload class, drive bin-packing and co-location to close it and make the weekly allocation review own the number. Below the band we shrink reservations; above it we add capacity ahead of the launch.