2 min lesson
Compute-optimal is not the same as deployment-optimal
Give a practical answer to this. "You have a fixed training-compute budget and your model will be served to millions of users for years. Why might you choose a model smaller than Chinchilla-optimal and train it on extra tokens?"
Step 1 of 2
Compute-optimal is not the same as deployment-optimalthe inference twist
Chinchilla minimizes training loss per training FLOP. At Cursor's volume, the bill that dominates is inference. That flips the calculus: a smaller model trained on far more tokens than Chinchilla-optimal can match a bigger one's quality while costing much less to serve per token.
- Objective
- Compute-optimal (Chinchilla)
- What you optimize
- Loss per training FLOP
- Tends to favor
- Balanced N and D
- Objective
- Deployment-optimal
- What you optimize
- Total cost incl. serving
- Tends to favor
- Smaller N, much larger D ('over-train')
- Objective
- Latency-critical (Cursor TabCursor's original autocomplete: multi-line, edit-aware suggestions you accept with the Tab key. Press Enter for the full definition.)
- What you optimize
- Tokens/sec at quality bar
- Tends to favor
- Smaller, heavily distilled or over-trained model
| Objective | What you optimize | Tends to favor |
|---|---|---|
| Compute-optimal (Chinchilla) | Loss per training FLOP | Balanced N and D |
| Deployment-optimal | Total cost incl. serving | Smaller N, much larger D ('over-train') |
| Latency-critical (Cursor TabCursor's original autocomplete: multi-line, edit-aware suggestions you accept with the Tab key. Press Enter for the full definition.) | Tokens/sec at quality bar | Smaller, heavily distilled or over-trained model |
The right scaling target depends on whether training FLOPs or lifetime serving FLOPs dominate the bill.
Name where the law stops applying. Say: "Chinchilla is compute-optimal for training, but for a product served at Cursor's volume I'd deliberately over-train a smaller model, because inference FLOPs over its lifetime swamp the one-time training cost. The scaling exponent still tells me how much quality I'm trading away." That nuance is what shows you have shipped a model and not only read the paper.
Don't treat scaling laws as exact prophecy. They hold within a regime and a data distribution; data quality, repetition past a few epochs and architecture changes all bend the curve. Call them a planning tool with error bars. Data-limited settings, like a finite pool of high-quality code, break the clean story.