GPU Economics & Capacity Planning

A GPU costs roughly the same whether it runs at 90% or 9%. That one fact reshapes how you think about every dollar of Cursor's inference bill.

Inference is the dominant COGS line at a company serving 1M+ developers and the asset behind that line is a fleet of accelerators that are expensive to acquire, hungry for power and impossible to pause. Unlike a CPU box you can scale to zero, a reserved H100 bills the same per hour at idle as under full load. So the real cost question is never just "how many GPUs" - it is "how busy are the ones we already pay for."

When a systems-literacy interviewer asks what drives GPU cost, they want the cost structure, not a list of part numbers. There are four levers and idle time is the one that quietly wastes the most.

Reserved vs. on-demand vs. serverlessthe buy decision is a bet on demand certainty

The cheapest hour comes from committing ahead of time and the most flexible hour comes from buying it the moment you need it. Choosing the mix is a bet on how confident you are in the forecast.

The number that ties this together is utilization. Reserved capacity only beats on-demand if you actually keep it busy. Run a committed cluster at 20% and you have paid the low per-hour rate on mostly dead hours, which is often worse than on-demand at the volume you really used.

GPUs vs. other accelerators

You do not need to design silicon, but you should reason about the rough fit by workload. Training and large-context prefill are throughput jobs that reward raw compute; tens-of-millisecond inference is a latency job where the constraint is response time, not aggregate FLOPs.

Cost per inference is not a property of the model. It falls out of four serving decisions made together: model size, utilization, batching and routing.

On the systems round you will be expected to reason fluently about serving without writing kernels. The mechanism that ties throughput to cost is batching: the GPU runs many requests through one forward pass, so the fixed cost of that pass is shared. Bigger batches mean lower cost per request. They also mean a request may wait for the batch to fill, which adds latency. That tension is the heart of inference serving.

Latency is a distribution, not an averagethe tail is what users feel

An average hides the requests that actually hurt. If p50 is 40ms but p99 is 600ms, one in a hundred completions feels broken and a developer hits that tail many times an hour. Manage latency as percentiles and write SLAs against the tail.

Model routingsend the easy work to the cheap model

Not every request deserves the biggest model. Model routing sends simple completions to a small, cheap, fast model and reserves the large model for genuinely hard queries. Done well it cuts cost and latency at the same time, because the small model is both cheaper to run and quicker to respond. The risk is misrouting: a hard query sent to the small model returns a weak answer, so the router's accuracy is itself a quality lever.

1. 1Classify. A lightweight signal estimates query difficulty before committing GPU time.
2. 2Route. Simple queries go to the small model; hard ones go to the large model.
3. 3Measure. Track quality and cost per route, not just blended averages, so misrouting is visible.
4. 4Tune. Shift the threshold as model capabilities and demand change - this is never set-and-forget.

GPUs in the wild commonly sit at 15-30% utilization. Closing the gap to 60-80% is the single largest efficiency program available to an infra TPM.

If a GPU costs the same idle or busy, then utilization is efficiency. The reason the number is usually low is rarely one big mistake - it is fragmentation, over-reservation and bursty demand each shaving off a slice. Your job is to make utilization a managed metric with a target band, not a thing people glance at after the fact.

The levers that raise itstandard moves, in order of usual payoff

That last card surprises people. A 10% failure-and-retry rate is not only a reliability problem - it means a tenth of your GPU-seconds produced nothing, then got spent again on the retry. Failed work is doubly expensive and it hides inside utilization that looks busy.

A capacity forecast that starts from "how many GPUs do we want" is guessing. A credible one starts from product demand and derives the GPUs.

The chain runs from users to compute. Demand signals drive request volume, request volume and model choice drive GPU-seconds and GPU-seconds plus a utilization target drive the fleet size. Build it in that order and you can defend every number to ML and Finance, because each step is an assumption someone can challenge.

1. 1Start from demand signals. DAU, requests per active user and the rollout schedule of new models and features - not a GPU count pulled from last quarter.
2. 2Translate to compute. Map request volume and model/routing mix to GPU-seconds, stating the cost-per-request assumptions explicitly.
3. 3Split baseline from peak. Model the steady reserve separately from burst; do not provision the whole fleet for the spikiest hour.
4. 4Apply a utilization target. Divide required compute by the target band to get fleet size, so the plan bakes in efficiency.
5. 5Run scenarios. Build base, upside and downside and pre-decide the trigger points that move you between them.

Baseline vs. peakprovisioning for peak everywhere is how budgets blow up

Lead time and the cost of being wrongGPU supply is not instant

GPUs cannot be conjured on the day you need them. Supply is constrained and lead times are real, so a forecast has to run ahead of demand by the acquisition window or the plan is fiction. That lead time is exactly why scenario planning earns its keep: you pre-commit trigger points so you are ordering capacity before the upside case arrives, not after.

The discipline that makes a forecast trustworthy is quantifying the cost of being wrong in each direction. Over-provision and you strand reserved spend on idle cards. Under-provision and a launch is capacity-starved, degrading latency or blocking growth. Those costs are rarely symmetric and naming which way you are biased - and why - is what a senior leader is listening for.

Capacity decisions are not made in a spreadsheet. They are made in a room where ML and Infra leaders trade off whose work gets the GPUs - and the TPM owns that room.

This is where the role's whole thesis lands: influence without authority. You do not get to assign GPUs by fiat and you should not want to. You build the framework, run the forum and own the data, so the decision is made on evidence by the people accountable for it. The final political call belongs to leadership; the quality of the call is yours.

1. 1Stand up the cadence. A recurring allocation review with ML and Infra leadership and one named decision owner - not a committee that diffuses accountability.
2. 2Require evidence to request. Every capacity ask brings current utilization, projected demand and expected value; no narrative-only requests.
3. 3Make tradeoffs explicit. Funding one team's GPUs is declining another's - surface the opportunity cost in the room, do not let it stay implicit.
4. 4Decide and record. The owner calls it; you log the decision and its rationale in a living ledger.
5. 5Revisit on real data. Next cycle, check whether the granted capacity got used as promised and claw back what did not.

The bar to request capacityno evidence, no allocation

The capacity ledgerdecisions you can audit and revisit

Allocation drifts when grants are made and never revisited. A living capacity ledger records who got what, on what evidence and whether the bet paid off. It makes the forum self-correcting: a team that consistently under-uses its grant loses the argument next cycle on data, not politics. It also gives you the honest input for COGS attribution, because you can connect spend back to the team and product that asked for it.