1 min lesson
GPUs vs. other accelerators
Use "GPUs vs. other accelerators" to tell the cases apart, then choose a response for each one.
Step 1 of 2
GPUs vs. other accelerators
You do not need to design silicon, but you should reason about the rough fit by workload. Training and large-context prefill are throughput jobs that reward raw compute; tens-of-millisecond inference is a latency job where the constraint is response time, not aggregate FLOPs.
Long, throughput-bound jobs
Top-end interconnect and memory matter
Lives in R&D spend, not production COGS
Latency-tolerant, can batch hard
Optimize for cost per request
Highest utilization is achievable here
Tens-of-ms budgets (tab prediction)
Batching headroom is tight
You pay for responsiveness, not throughput
Do not conflate cheaper-per-hour with cheaper-overall. The trap question is "reserved is discounted, so we should reserve everything, right?" The honest answer is no: reserving capacity you cannot keep busy converts a flexible cost into stranded spend. Anchor on effective cost per useful computation, not the sticker per-hour rate.