2 min lesson
GPU economics and provider model
Use "GPU cost is mostly about keeping expensive silicon busy" to describe the practical rule.
Step 1 of 3
Everything in the previous four sections eventually becomes a number on a bill or a capacity plan. The routing engineer who can do this math is the one who earns the gray-area decisions.
GPU cost is mostly about keeping expensive silicon busy. Utilization is driven by batch size and how well the KV cache fits in memory: idle GPUs and tiny batches burn rent for nothing, while a well-packed batch spreads the fixed cost of loading weights across many requests. The mechanics from earlier sections - paged KV, continuous batching, prefix-cache hits - are the same levers that move utilization.
Learn more
Full explanation
From Silicon to the Dollar
Interactive diagram. Tab through its regions; each focused region shows its detail in the panel below.
Each layer rests on the one below: cost is set at the bottom, then earned back by how well the layers above pack it.
- Per-token pricing
- Separate input (prefill) and output (decode) rates; output is usually pricier
- Rate limits
- RPM (requests/min) and TPM (tokens/min) caps per key/account
- Reserved vs on-demand
- Committed/provisioned capacity (cheaper, fixed) vs on-demand (flexible, pricier, throttled)
- GPU utilization
- Driven by batch size and KV-cache fit - the core efficiency metric for self-hosting
- Cache hit rate
- Fraction of prefix work served from cache - a first-class cost metric, not a footnote