1 min lesson
Utilization as the efficiency lever
Think through this situation: "A dashboard shows the GPU cluster at 96% utilization, yet the inference team says they constantly run out of capacity for new work. What is likely going on?" Give the practical answer in plain words.
Step 1 of 2
GPUs in the wild commonly sit at 15-30% utilization. Closing the gap to 60-80% is the single largest efficiency program available to an infra TPM.
If a GPU costs the same idle or busy, then utilization is efficiency. The reason the number is usually low is rarely one big mistake - it is fragmentation, over-reservation and bursty demand each shaving off a slice. Your job is to make utilization a managed metric with a target band, not a thing people glance at after the fact.
Allocation utilization is how much capacity is reserved or assigned to teams. Compute utilization is how busy those GPUs actually are. A cluster can be 100% allocated and 20% busy - every card has an owner, almost none are working. The dangerous metric is allocation, because it looks full while the money leaks. Always ask which one a dashboard is showing.