2 min lesson
Scaling laws and compute-optimal training
Take this situation: "Training compute is fixed at C ≈ 6ND. You halve N (the parameter count) while holding C constant. Roughly how much more training data D fits in the same budget?" Lead with your decision, then add the reason.
Step 1 of 3
"Train with less compute" is in the job description and scaling laws are the language for spending a fixed FLOP budget well. Expect to be asked, on a whiteboard, how you'd allocate a budget between a bigger model and more data.
The headline result is Chinchilla: for a fixed training-compute budget, loss is minimized by scaling parameters and training tokens together, roughly in proportion, rather than pouring everything into a giant under-trained model. The earlier GPT-3-era models were large and badly under-trained on data.