Training-time scaling: why bigger, longer, and more data kept working
For years the recipe for a better model was blunt: more parameters, more data, more compute. Scaling laws made that predictable — and eventually expensive. Here's the idea and its limits.
The defining discovery of the last decade of AI is almost embarrassingly simple: make the model bigger, feed it more data, and spend more compute training it, and it reliably gets better. Not by magic — by a smooth, measurable curve. That's training-time scaling, and it's why models went from novelty to frontier in a few years.
Scaling laws: performance you can predict
Researchers found that a model's loss falls predictably as you scale three things together: parameters, training data, and compute. The relationship is smooth enough that you can forecast how good a bigger model will be before you train it — which is exactly what turns model-building from alchemy into engineering.
The three dials
- Parameters — more capacity to represent patterns.
- Data — more (and higher-quality) tokens to learn from.
- Compute — more training to actually fit all those parameters to all that data.
The catch is they have to grow in balance. A huge model trained on too little data is undertrained; the Chinchilla work showed many big models were, and that a smaller model trained on more data could beat them. Bigger isn't the goal — bigger-and-balanced is.
Where it runs out
Scaling laws are smooth, but the costs behind them aren't. Each increment of quality takes a larger multiple of compute, data, and money than the last. Frontier training runs now cost enormous sums, high-quality data is finite, and the curve keeps flattening. Training-time scaling still works — it's just no longer the only, or the cheapest, lever.
Scaling laws turned 'will a bigger model be better?' from a hope into a forecast. The new question is whether the forecast is worth the invoice.