1 min lesson
Prefill vs decode
Tell someone how to act on this idea: "Once you see the split, most latency and cost numbers in the deep-dive stop being mysterious."
Step 1 of 2
Every LLM request runs in two phases with completely different physics. Once you see the split, most latency and cost numbers in the deep-dive stop being mysterious.
When a request hits a model, the engine first reads the entire prompt at once, then emits new tokens one by one. The first step is prefill. The second is decode. They stress different parts of the GPU, so they show up as different numbers on your dashboards and different lines on the provider bill.
Prefill takes the whole prompt and runs it through the model in a single forward pass that fills the attention cache and produces exactly one token: the first one. Because every prompt token is processed in parallel, this phase is compute-bound - it saturates the GPU's matrix-multiply units. It is what you wait on before anything appears on screen, so it sets the time-to-first-token (TTFT).
Decode then generates the rest, token by token, each new token attending to everything before it. There is no parallelism across the sequence here: token N must exist before token N+1 can be computed. The bottleneck is moving model weights and the cache through GPU memory, so this phase is memory-bandwidth-bound. It sets inter-token latency (ITL) and, multiplied across hundreds of output tokens, most of the total wall-clock time on a long answer.
Learn more
Full explanation
One Request, Two Phases
Interactive diagram. Step through it with the Next and Previous controls below, or Tab to a region to read its detail.
Step through a single request: prompt in, one prefill pass, then a decode loop that emits the rest.
Learn more
Advanced table
Two phases, two bottlenecks
- Property
- What it does
- Prefill
- Process the whole prompt, emit token 1
- Decode
- Emit each subsequent token autoregressively
- Property
- Parallelism
- Prefill
- All prompt tokens at once
- Decode
- Strictly sequential, one token per step
- Property
- Bottleneck
- Prefill
- Compute (matmul / FLOPs)
- Decode
- Memory bandwidth (weights + KV through HBM)
- Property
- Latency metric it drives
- Prefill
- TTFT (time-to-first-token)
- Decode
- ITL (inter-token latency)
- Property
- Scales with
- Prefill
- Prompt / context length
- Decode
- Number of output tokens
- Property
- Cost line it maps to
- Prefill
- Input-token pricing
- Decode
- Output-token pricing
| Property | Prefill | Decode |
|---|---|---|
| What it does | Process the whole prompt, emit token 1 | Emit each subsequent token autoregressively |
| Parallelism | All prompt tokens at once | Strictly sequential, one token per step |
| Bottleneck | Compute (matmul / FLOPs) | Memory bandwidth (weights + KV through HBM) |
| Latency metric it drives | TTFT (time-to-first-token) | ITL (inter-token latency) |
| Scales with | Prompt / context length | Number of output tokens |
| Cost line it maps to | Input-token pricing | Output-token pricing |
Two phases, two bottlenecks - internalize this table and the rest of the module follows.
Why a routing engineer caresthe editor surfaces feel these two numbers differently
Cursor's surfaces have wildly different latency profiles and that difference is really a prefill-vs-decode difference. Tab completion lives or dies on TTFT - the suggestion has to appear before the developer's fingers move on, inside a sub-100ms budget. A long Agent generation cares far more about sustained ITL and throughput, because the user is reading a stream and the total time is dominated by hundreds of decode steps.
Short prompt, short output
TTFT is everything (sub-100ms feel)
Prefill-sensitive, decode is tiny
Medium prompt, focused edit
TTFT still dominates perceived speed
A few hundred ms is acceptable
Large context, long output
ITL + throughput dominate total time
Prefill on big repo context can hurt TTFT too
Long prompts are where the two phases collide. Stuff a large repo or a 100k-token context into a request and prefill cost explodes, because it scales with prompt length. That is a routing and caching problem before it is a model problem: the same repo context resent on every keystroke is the single biggest lever you have. We will pull that lever in the next section.
In the deep-dive you should be able to draw, freehand, where time and money go: a short prompt with a long answer is decode-dominated (output tokens, ITL); a huge context with a one-line answer is prefill-dominated (input tokens, TTFT). Naming which phase dominates a given workload is the move that signals you actually understand inference rather than reciting terms.
Do not say a request is "slow because the model is big" and stop there. The interviewer wants the phase: is TTFT bad (prefill - long context, cold cache, queueing) or is ITL bad (decode - memory bandwidth, large batch contention)? The fix is different in each case and conflating them reads as surface knowledge.
Learn more
Optional practice
Practice: Prefill vs decode
QWhich statement about the two inference phases is correct?