1 min lesson
Mock inference deep-dive
Use "And it moves fast, so vague answers get caught immediately" to describe the practical rule.
Step 1 of 4
The design round tests whether you can shape the system. This round tests whether you understand the machine underneath it - and it moves fast, so vague answers get caught immediately.
Treat the rubric as a gap finder, not a score to admire. Mark the weakest proof, turn it into one practice rep and keep the artifact you would show an interviewer.
Have a partner read the rapid-fire prompts and stop you the instant you hand-wave. The hand-wave is the signal; mark it and study it.
Learn more
Advanced table
Term
- Term
- Prefill vs decode
- The 20-second answer
- Prefill processes the whole prompt in parallel (compute-bound, builds the KV cache). Decode generates one token at a time, reusing the cache (memory-bandwidth-bound).
- Term
- KV cache
- The 20-second answer
- Stored key/value tensors for past tokens so decode doesn't recompute attention over the whole prompt each step. Grows with context length; it's the memory pressure that limits batch size.
- Term
- Prefix caching
- The 20-second answer
- Reuse the KV cache for a shared prompt prefix across requests (system prompts, repo context). Turns repeat prefill into a cache hit; huge for Cursor's repeated context.
- Term
- Continuous batching
- The 20-second answer
- Add and evict requests from the running batch per decode step instead of waiting for the slowest to finish. Keeps the GPU saturated; the main lever for throughput.
- Term
- TTFT vs ITL
- The 20-second answer
- Time-to-first-token is dominated by prefill and queueing; inter-token latency is the decode cadence. Tab lives or dies on TTFT; long chat outputs feel slow on bad ITL.
- Term
- Provider economics
- The 20-second answer
- Per-token pricing differs for input vs output; rate limits cap your headroom; batching and cache hits are what move cost per request, not the sticker price.
| Term | The 20-second answer |
|---|---|
| Prefill vs decode | Prefill processes the whole prompt in parallel (compute-bound, builds the KV cache). Decode generates one token at a time, reusing the cache (memory-bandwidth-bound). |
| KV cache | Stored key/value tensors for past tokens so decode doesn't recompute attention over the whole prompt each step. Grows with context length; it's the memory pressure that limits batch size. |
| Prefix caching | Reuse the KV cache for a shared prompt prefix across requests (system prompts, repo context). Turns repeat prefill into a cache hit; huge for Cursor's repeated context. |
| Continuous batching | Add and evict requests from the running batch per decode step instead of waiting for the slowest to finish. Keeps the GPU saturated; the main lever for throughput. |
| TTFT vs ITL | Time-to-first-token is dominated by prefill and queueing; inter-token latency is the decode cadence. Tab lives or dies on TTFT; long chat outputs feel slow on bad ITL. |
| Provider economics | Per-token pricing differs for input vs output; rate limits cap your headroom; batching and cache hits are what move cost per request, not the sticker price. |