1 min lesson
The latency levers
Connect "The budget proves the design" to the decision you would make.
Step 1 of 3
The latency levers
The budget proves the design: a big model over a far network can't fit, so the levers below are forced, not optional.
Learn more
Advanced table
Lever
- Lever
- Debounce + cancel in-flight
- What it buys
- Kills most requests before they're sent - the user is still typing
- Cost / catch
- Tune the window: too long feels laggy, too short wastes calls
- Lever
- Small, specialized model
- What it buys
- The only way inference fits the budget at this volume
- Cost / catch
- Lower ceiling on hard completions - route those elsewhere
- Lever
- Speculative / early decoding
- What it buys
- Start emitting before the full sequence is decoded
- Cost / catch
- Wasted compute on speculations that don't pan out
- Lever
- Caching
- What it buys
- Repeat prefixes and contexts skip inference entirely
- Cost / catch
- Cache keys must include enough context to stay correct
- Lever
- Prefetch on pause
- What it buys
- Compute likely completions during a typing pause, serve instantly
- Cost / catch
- Speculative spend; bound it so it doesn't dominate cost
- Lever
- Edge proximity
- What it buys
- Cuts the fixed network tax that no model trick can recover
- Cost / catch
- Operational complexity of regional inference
| Lever | What it buys | Cost / catch |
|---|---|---|
| Debounce + cancel in-flight | Kills most requests before they're sent - the user is still typing | Tune the window: too long feels laggy, too short wastes calls |
| Small, specialized model | The only way inference fits the budget at this volume | Lower ceiling on hard completions - route those elsewhere |
| Speculative / early decoding | Start emitting before the full sequence is decoded | Wasted compute on speculations that don't pan out |
| Caching | Repeat prefixes and contexts skip inference entirely | Cache keys must include enough context to stay correct |
| Prefetch on pause | Compute likely completions during a typing pause, serve instantly | Speculative spend; bound it so it doesn't dominate cost |
| Edge proximity | Cuts the fixed network tax that no model trick can recover | Operational complexity of regional inference |