LLM Inference Fundamentals for the Routing Engineer

Every LLM request runs in two phases with completely different physics. Once you see the split, most latency and cost numbers in the deep-dive stop being mysterious.

When a request hits a model, the engine first reads the entire prompt at once, then emits new tokens one by one. The first step is prefill. The second is decode. They stress different parts of the GPU, so they show up as different numbers on your dashboards and different lines on the provider bill.

Prefill takes the whole prompt and runs it through the model in a single forward pass that fills the attention cache and produces exactly one token: the first one. Because every prompt token is processed in parallel, this phase is compute-bound - it saturates the GPU's matrix-multiply units. It is what you wait on before anything appears on screen, so it sets the time-to-first-token (TTFT).

Decode then generates the rest, token by token, each new token attending to everything before it. There is no parallelism across the sequence here: token N must exist before token N+1 can be computed. The bottleneck is moving model weights and the cache through GPU memory, so this phase is memory-bandwidth-bound. It sets inter-token latency (ITL) and, multiplied across hundreds of output tokens, most of the total wall-clock time on a long answer.

Why a routing engineer caresthe editor surfaces feel these two numbers differently

Cursor's surfaces have wildly different latency profiles and that difference is really a prefill-vs-decode difference. Tab completion lives or dies on TTFT - the suggestion has to appear before the developer's fingers move on, inside a sub-100ms budget. A long Agent generation cares far more about sustained ITL and throughput, because the user is reading a stream and the total time is dominated by hundreds of decode steps.

Long prompts are where the two phases collide. Stuff a large repo or a 100k-token context into a request and prefill cost explodes, because it scales with prompt length. That is a routing and caching problem before it is a model problem: the same repo context resent on every keystroke is the single biggest lever you have. We will pull that lever in the next section.

The KV cache is the single most important object in inference economics. It is why decode is fast, it is your tightest memory constraint and prefix caching is the biggest cost lever Cursor has.

Attention needs the keys and values of every previous token to compute the next one. Without a cache, each new token would re-run attention over the entire sequence from scratch - quadratic work that would make decode hopelessly slow. The KV cache stores those keys and values per token, so generating token N is cheap: attend to the cached state plus the one new token.

That speed has a price in memory. The cache grows linearly with context length and with how many requests share the GPU at once.

Because the cache is the capacity ceiling, how you manage its memory directly sets how many concurrent users a GPU can hold. Naive allocation reserves the full context window per request up front, which wastes huge amounts of memory on requests that finish short. PagedAttention, introduced by vLLM, fixes this by chopping the cache into fixed-size pages allocated on demand, the way an OS pages physical memory. Fragmentation drops, utilization rises and effective batch size goes up without buying more GPUs.

Prefix caching: reuse the work you already didthe lever with Cursor's name on it

If two requests start with the same tokens, their KV cache for that shared span is identical. Prefix caching keeps those computed keys/values around and reuses them, so a request that shares a prefix skips prefilling it entirely. The first matching token is then the only new prefill work.

Cursor's prompts are unusually repetitive, which is exactly the condition prefix caching loves. The same system prompt, the same tool definitions and often the same file or repo context get sent across many requests in a session. A high prefix-cache hit rate turns an expensive full prefill into a near-free cache read.

Batching is how a GPU serves many users at once instead of one. How you batch is the throughput-versus-latency knob at the heart of serving - name it explicitly and the design round opens up.

GPUs are fastest when they process many sequences together, amortizing the cost of loading weights across the whole batch. The question is when you form the batch and what you do when sequences in it finish at different times.

The win from continuous batching is that the GPU never idles waiting for the slowest sequence and a short request stuck behind a long one doesn't wait for the long one to finish. Each decode step re-forms the working set, so capacity tracks demand step by step.

The tradeoff you must name out loudbigger batches are not free

Pushing batch size up raises throughput and GPU utilization - more tokens per dollar. It also raises per-request and tail latency, because each request now shares memory bandwidth and compute with more neighbors and the decode step for a larger batch takes longer. This is the throughput/latency tradeoff and serving research (vLLM, Sarathi-Serve) is largely about pushing what's new of it.

Sarathi-Serve's contribution is worth a sentence: it chunks long prefills and interleaves them with ongoing decodes so a big prefill doesn't stall everyone's token stream. The general principle is that prefill and decode compete for the same GPU and how you schedule them against each other shapes tail latency.

What this means for provider-backed routingyou don't own their batcher, but you own the dials in front of it

When you route to a third-party provider, their serving stack does the batching and you can't see inside it. What you control is upstream: concurrency limits, queueing and admission, hedging and which provider gets a given request. Those decisions interact with the provider's batch composition even though you never touch it.

A gateway that buffers a full response before sending it has already lost. Streaming, token accounting and cancellation are where inference mechanics become concrete gateway code.

Models emit output token by token, almost always over Server-Sent Events (SSE). The whole point is that the user sees text appear as it's generated. A gateway must pass those chunks through as they arrive, not collect the full answer and forward it at the end - buffering would throw away the streaming UX and inflate perceived latency to total latency.

1. 1Open upstream as a stream. Request SSE from the provider and read the response body incrementally, never await the whole thing into memory.
2. 2Flush each chunk downstream. Forward provider events to the client as they land, preserving event boundaries; do not coalesce or reorder.
3. 3Measure both clocks. Record TTFT at the first token and total latency at stream end - they're different numbers and users feel the first one.
4. 4Propagate completion and errors. Send the terminal event on success and on an upstream error mid-stream, surface it cleanly rather than hanging the connection.
5. 5Wire cancellation through. If the client disconnects or aborts, abort the upstream request so you stop generating (and paying for) tokens nobody will read.

// AbortController lets a client cancel propagate to the provider.
async function proxyStream(req: Request, provider: Provider): Promise<Response> {
  const ac = new AbortController();
  req.signal.addEventListener("abort", () => ac.abort()); // client gone -> stop upstream

  const upstream = await provider.fetch({ signal: ac.signal, stream: true });
  if (!upstream.body) throw new GatewayError("no stream body");

  let firstToken = false;
  const t0 = performance.now();
  const out = upstream.body.pipeThrough(
    new TransformStream({
      transform(chunk, controller) {
        if (!firstToken) { firstToken = true; recordTTFT(performance.now() - t0); }
        controller.enqueue(chunk); // flush immediately, no buffering
      },
    }),
  );
  return new Response(out, { headers: { "content-type": "text/event-stream" } });
}

Tokens are not characters and they're not portablecounting is per-provider

Tokenization differs by model and provider - the same string is a different token count under different tokenizers. Token counts drive three things you can't fudge: cost estimates (pricing is per token), context-limit checks (does this request fit?) and rate-limit accounting (TPM budgets). A gateway that estimates tokens with the wrong tokenizer will misprice requests and either reject valid ones or let oversized ones through.

Context windows are finite, so the gateway needs a graceful answer when a request won't fit. Two honest options: reject early with a clear error or down-route to a model with a bigger window (and a different price). Silently truncating the prompt is the wrong move - it changes the user's request without telling them.

Everything in the previous four sections eventually becomes a number on a bill or a capacity plan. The routing engineer who can do this math is the one who earns the gray-area decisions.

GPU cost is mostly about keeping expensive silicon busy. Utilization is driven by batch size and how well the KV cache fits in memory: idle GPUs and tiny batches burn rent for nothing, while a well-packed batch spreads the fixed cost of loading weights across many requests. The mechanics from earlier sections - paged KV, continuous batching, prefix-cache hits - are the same levers that move utilization.

Self-hosted vs provider-hostedthe tradeoff that frames the whole role

This is the strategic fork the inference platform lives on. Hosting your own models on rented or owned GPUs gives you control and the best cost-at-scale, at the price of running a serving stack, capacity planning and on-call. Calling a provider's API gives you flexibility and operational simplicity, at the price of per-token margin, rate limits and being exposed to their outages. A mature platform does both and routes between them.

Capacity planning, back-of-the-envelopefind the bottleneck before you build

The deep-dive often ends with a sizing question. The method is to estimate demand in the same units as your limits, then see what hits the ceiling first.

1. 1Estimate demand. Peak QPS × average tokens per request (input + output) gives you tokens/sec and requests/sec at peak.
2. 2Compare to provider limits. Check that demand against RPM and TPM caps - TPM usually binds first for long-context traffic.
3. 3Compare to GPU memory. For self-hosting, peak concurrency × KV per request must fit in HBM alongside weights or batch size collapses.
4. 4Name the binding constraint. Say which ceiling you hit first (TPM, RPM or KV memory) - that's the one your design has to relieve.