2 min lesson
Compute-optimal is not the same as deployment-optimal
Give a practical answer to this: "You have a fixed training-compute budget and your model will be served to millions of users for years. Why might you choose a model smaller than Chinchilla-optimal and train it on extra tokens?"
Step 1 of 2
Compute-optimal is not the same as deployment-optimalthe inference twist
Chinchilla minimizes training loss per training FLOP. Cursor serves ComposerCursor's own fast coding model, tuned for the editor and priced well below frontier models; the recommended day-to-day model for executing a plan. Press Enter for the full definition. to millions of sessions, so the bill that dominates is inference. That flips the calculus: a smaller model trained on far more tokens than Chinchilla-optimal can match a bigger one's quality while costing much less to serve per token.
- Objective
- Compute-optimal (Chinchilla)
- What you optimize
- Loss per training FLOP
- Tends to favor
- Balanced N and D
- Objective
- Deployment-optimal
- What you optimize
- Total cost incl. serving
- Tends to favor
- Smaller N, much larger D ('over-train')
- Objective
- Latency-critical (Cursor TabCursor's original autocomplete: multi-line, edit-aware suggestions you accept with the Tab key. Press Enter for the full definition.)
- What you optimize
- Tokens/sec at quality bar
- Tends to favor
- Smaller, heavily distilled or over-trained model
| Objective | What you optimize | Tends to favor |
|---|---|---|
| Compute-optimal (Chinchilla) | Loss per training FLOP | Balanced N and D |
| Deployment-optimal | Total cost incl. serving | Smaller N, much larger D ('over-train') |
| Latency-critical (Cursor TabCursor's original autocomplete: multi-line, edit-aware suggestions you accept with the Tab key. Press Enter for the full definition.) | Tokens/sec at quality bar | Smaller, heavily distilled or over-trained model |
The right scaling target depends on whether training FLOPs or lifetime serving FLOPs dominate the bill.
Show you know the limits of the law, not just the law. Say: "Chinchilla is compute-optimal for training, but for a product served at Cursor's volume I'd deliberately over-train a smaller model, because inference FLOPs over its lifetime swamp the one-time training cost. The scaling exponent still tells me how much quality I'm trading away." That nuance separates someone who read the paper from someone who ships.
Don't treat scaling laws as exact prophecy. They hold within a regime and a data distribution; data quality, repetition past a few epochs and architecture changes all bend the curve. The honest framing is that scaling laws are a planning tool with error bars, not a guarantee - and that data-limited settings (a finite pool of high-quality code) break the clean story.