Reading LLM benchmarks: MMLU, GPQA, AIME, LiveCodeBench, MuSR, HLE

Every model announcement comes with a table of benchmark scores, each one an acronym you're expected to already know. They're useful — but only if you know what each measures and how it can mislead. Here's a field guide to the ones you'll see most.

The benchmarks, briefly

• MMLU — Massive Multitask Language Understanding: multiple-choice questions across 57 subjects. The long-time general-knowledge yardstick, now largely saturated at the top, where everyone scores similarly high.
• GPQA — Graduate-level, 'Google-proof' science questions written by experts; the 'Diamond' subset is the hard, cleanly-verified core. Tests deep reasoning, not recall.
• AIME — the American Invitational Mathematics Examination: hard competition maths, a favourite for stress-testing reasoning models.
• LiveCodeBench — coding problems collected continuously over time to resist contamination, measuring real programming ability rather than memorised solutions.
• MuSR — Multistep Soft Reasoning: reasoning over long, natural-language narratives; rewards multi-hop thinking over pattern-matching.
• HLE (Humanity's Last Exam) — thousands of extremely hard expert questions across many domains, deliberately built to stay far from saturation as models improve.

How to read them without being sold to

• Contamination — if a benchmark's questions leaked into training data, the score measures memorisation, not ability. This is why contamination-resistant benchmarks exist.
• Saturation — when every model scores ~90%+, the benchmark has stopped discriminating; small gaps are noise.
• Relevance gap — a great MMLU score says little about how a model handles your support tickets or your codebase.
• Margins matter — a one- or two-point difference is often within noise, not a real capability gap.

Benchmarks tell you how a model does on someone else's exam. The only score that predicts your product is your eval on your data.