Prefill vs decode

Every LLM request runs in two phases with completely different physics. Once you see the split, most latency and cost numbers in the deep-dive stop being mysterious.

When a request hits a model, the engine first reads the entire prompt at once, then emits new tokens one by one. The first step is prefill. The second is decode. They stress different parts of the GPU, so they show up as different numbers on your dashboards and different lines on the provider bill.

Prefill takes the whole prompt and runs it through the model in a single forward pass that fills the attention cache and produces exactly one token: the first one. Because every prompt token is processed in parallel, this phase is compute-bound. It saturates the GPU's matrix-multiply units. It is what you wait on before anything appears on screen, so it sets the time-to-first-token (TTFT).

Decode then generates the rest, token by token, each new token attending to everything before it. There is no parallelism across the sequence here: token N must exist before token N+1 can be computed. The bottleneck is moving model weights and the cache through GPU memory, so this phase is memory-bandwidth-bound. It sets inter-token latency (ITL) and, multiplied across hundreds of output tokens, most of the total wall-clock time on a long answer.