The Chinchilla scaling law: the 20-tokens-per-parameter rule of thumb

Training-time scaling says bigger, more data, more compute makes better models (see that post). Chinchilla added the crucial refinement: for a fixed compute budget, how should you split it between model size and training data? The answer overturned how the field had been building models.

The finding

DeepMind found that the biggest models of the era — Gopher, GPT-3 and friends — were badly undertrained. Given their size, they'd been fed far too little data. To prove it, they trained Chinchilla: at 70B parameters, less than a third of Gopher's 280B, but on far more data — and it beat Gopher across the board, using the same compute.

The rule of thumb

Compute-optimal training uses roughly 20 training tokens for every model parameter — scale the two together, not the model alone.

That ratio is the takeaway most people carry: a 10B-parameter model wants on the order of 200B tokens to be compute-optimal. Scale parameters and data in balance, and you get more capability per dollar of compute than by simply inflating the parameter count.

• Bigger isn't better if it's undertrained — a smaller, well-fed model beats a larger, starved one.
• Data is a first-class lever, not an afterthought to parameter count.
• Compute has an optimal split between size and tokens — Chinchilla named it.

The nuance today

Chinchilla optimises training compute — but not inference. A model you'll serve billions of times is often deliberately 'overtrained': made smaller than Chinchilla-optimal and fed even more data, so it's cheaper to run forever after. That's why today's small models are trained on trillions of tokens, well past the rule of thumb. Chinchilla is the baseline you reason from, not a law you never break.

Chinchilla's gift wasn't a bigger model — it was permission to build a smaller one, and feed it properly.