1 min lesson
Cost control: route by difficulty
Take "Cost control: route by difficulty" step by step, then finish with the result that proves it worked.
Step 1 of 2
Cost control: route by difficulty
At this scale, cost and latency are the same conversation. You can't afford a big model on every keystroke and you don't need one. Most completions are trivial.
- 1Default to small. Send the overwhelming majority of completions to a fast, cheap model that handles the easy 90%.
- 2Detect difficulty. Use signals - low model confidence, a complex multi-line context, an explicit larger request - to flag the cases the small model will fumble.
- 3Escalate selectively. Route only the hard minority to a larger model, accepting its latency where the user expects a beat to think.
- 4Cache the wins. Memoize completions keyed on context so a repeated situation never pays inference twice.
The cascade
A small-model-first cascade with selective escalation is the answer that shows you understand product economics. It keeps p50 latency tiny and per-completion cost near zero and it spends the expensive model only where it changes the outcome. "Use a smaller model" is a fragment; "route easy to small, escalate hard, cache both" is a system.