Skip to lesson
Exit
LLM Inference Fundamentals for the Routing Engineer1 / 2

2 min lesson

Capacity planning, back-of-the-envelope

Talk this through in your own words: "You're asked to capacity-plan a new high-volume, long-context route. Walk through how you'd find the binding constraint." Finish with the next move.

Step 1 of 2

Capacity planning, back-of-the-envelopefind the bottleneck before you build

The deep-dive often ends with a sizing question. The method is to estimate demand in the same units as your limits, then see what hits the ceiling first.

  1. 1Estimate demand. Peak QPS × average tokens per request (input + output) gives you tokens/sec and requests/sec at peak.
  2. 2Compare to provider limits. Check that demand against RPM and TPM caps - TPM usually binds first for long-context traffic.
  3. 3Compare to GPU memory. For self-hosting, peak concurrency × KV per request must fit in HBM alongside weights or batch size collapses.
  4. 4Name the binding constraint. Say which ceiling you hit first (TPM, RPM or KV memory) - that's the one your design has to relieve.
Cache hit rate is a money metric

Because prefill is paid per input token, a few points of prefix-cache hit rate move the bill materially at Cursor's volume - the same system prompts and file contexts recur constantly. Treating cache hit rate as a first-class SLI you route to improve, not an implementation detail, is exactly the cost-at-scale fluency the JD asks for.

Say it like this

"I'd size against TPM first since our long-context Agent traffic is token-heavy, then check KV-cache memory for the self-hosted tier. My first cost lever is prefix-cache hit rate via affinity routing, because prefill on repeated repo context is where the input-token spend concentrates. Self-hosted for the steady high-volume models where utilization stays high, provider APIs for spiky or new models - and failover between them so no single backend outage is user-visible."

Watch out

Don't optimize utilization into a latency violation. Cranking batch size to push GPU utilization toward 100% is a great way to blow the Tab p99 budget. The right framing is "highest utilization that still meets the latency SLO," with the answer differing by surface - high-utilization batches for bulk Agent work, leaner batches for latency-critical Tab.

Learn more

Optional practice

Practice: Capacity planning, back-of-the-envelope

QWhy is prefix-cache hit rate treated as a first-class cost metric rather than an implementation detail?