How an LLM is trained, end to end: from raw data to a fine-tuned model

The chat model you call through an API is the last step of a long pipeline. Knowing the whole journey — from a pile of raw text to a helpful assistant — demystifies a lot of the field and makes it obvious where the cost, the effort, and the quality actually come from.

1. Curate the data

A model is its data. This stage collects a massive, diverse corpus — web text, books, code — and, just as importantly, filters it hard: dedup, quality-filter, and strip the junk. At trillion-token scale, 'garbage in, garbage out' is the whole ballgame, and data curation is where a surprising amount of a model's quality is decided.

2. Pre-process and tokenise

The cleaned text is normalised and tokenised into the integer ids the model trains on (see the tokens post). Unglamorous data engineering — but it's the majority of the actual work, and mistakes here quietly poison everything downstream.

3. Pretrain a base model

The expensive part: train a transformer to predict the next token across the entire corpus, adjusting billions of parameters via gradient descent. What comes out is a base model — fluent and knowledgeable, but raw: it completes text rather than following instructions (see the model-types post).

4. Scale toward the frontier

Bigger networks, more data, and more compute — kept in balance per the scaling laws (and Chinchilla) — push the base model toward frontier capability. This is where the GPUs and the money go, and where training-time scaling plays out.

5. Fine-tune into an assistant

Finally, the raw predictor is turned into something helpful: supervised fine-tuning on demonstrations, then alignment with RLHF (see the self-attention and RLHF post). Teams then optionally fine-tune further on their own domain — which is where techniques like QLoRA come in.

A frontier model isn't one breakthrough — it's a disciplined pipeline where curated data and careful fine-tuning matter as much as the compute in the middle.