2 min lesson
Prompt length drives input tokens and time to first token
In "Prompt length drives input tokens and time to first token", choose the best answer to "Which statement about the two inference phases is correct?" Then say what led you there.
Step 1 of 3
- Property
- Bottleneck
- Prefill
- Compute (matmul / FLOPs)
- Decode
- Memory bandwidth (weights + KV through HBM)
- Property
- Latency metric it drives
- Prefill
- TTFT (time-to-first-token)
- Decode
- ITL (inter-token latency)
- Property
- Scales with
- Prefill
- Prompt / context length
- Decode
- Number of output tokens
- Property
- Cost line it maps to
- Prefill
- Input-token pricing
- Decode
- Output-token pricing
| Property | Prefill | Decode |
|---|---|---|
| Bottleneck | Compute (matmul / FLOPs) | Memory bandwidth (weights + KV through HBM) |
| Latency metric it drives | TTFT (time-to-first-token) | ITL (inter-token latency) |
| Scales with | Prompt / context length | Number of output tokens |
| Cost line it maps to | Input-token pricing | Output-token pricing |
Prompt length is an input-token and TTFT problem; output length is an output-token and ITL problem.
Learn more
Full explanation
Sketch the time/dollar split on demand
In the deep-dive you should be able to draw, freehand, where time and money go. A short prompt with a long answer is decode-dominated, so the spend sits in output tokens and ITL. A huge context with a one-line answer is prefill-dominated, so it sits in input tokens and TTFT. Naming which phase dominates a given workload is the move that signals you actually understand inference rather than reciting terms.
Do not say a request is "slow because the model is big" and stop there. The interviewer wants the phase. Is TTFT bad? That points at prefill: long context, a cold cache or queueing. Is ITL bad? That points at decode: memory bandwidth or contention in a large batch. The fix is different in each case, and conflating them reads as surface knowledge.