Classical NLP foundations still worth knowing: TF-IDF, BM25, clustering, and metrics
LLMs didn't make classical NLP obsolete — hybrid search, deduplication, and evaluation all rest on it. A refresher on the pre-transformer toolkit that still quietly runs production systems.
The transformer era buried a lot of classical NLP in the conversation — but not in production. BM25 is half of every hybrid search, clustering organises document sets, NER feeds knowledge graphs, and classical metrics judge all of it. If your background includes this, it's an asset; if it doesn't, here's the refresher that still pays off.
TF-IDF and BM25: ranking by words
TF-IDF weights a term by how often it appears in a document (term frequency) against how rare it is across the corpus (inverse document frequency) — common words count for little, distinctive ones for a lot. BM25 refines this with saturation (each extra occurrence of a term helps less than the last) and length normalisation (a long document doesn't rank highly just for being long). BM25 is still the lexical half of hybrid search in the advanced-RAG post — decades old and not going anywhere.
Classification and clustering
- Document classification — assign labels: spam or not, topic, sentiment, routing category.
- Clustering — group without labels: k-means (fast, but you must pick k) and HDBSCAN (density-based, finds the number of clusters and flags outliers as noise).
- Topic modeling — techniques like LDA surface recurring themes across a corpus without supervision.
NER and deduplication
Named-Entity Recognition extracts structured entities — people, organisations, technical terms — from raw text; it's the classical cousin of the entity-extraction step in GraphRAG. Near-duplicate detection (MinHash and LSH, or cosine similarity over embeddings) finds and removes redundant documents, which matters enormously: duplicates poison training data and clutter retrieval results alike.
The metrics that judge everything
Precision, recall, and their harmonic mean F1; the ROC curve and its AUC for threshold-independent quality; and calibration — whether a model's stated confidence matches how often it's actually right. These aren't legacy trivia: recall@k for retrieval and precision/recall for classification are exactly how you evaluate an LLM system today (see the eval posts). The vocabulary of evaluation is classical, and teams fluent in it debug faster.
LLMs sit on top of classical NLP, not in place of it. BM25 still ranks, k-means still clusters, and F1 still judges — the foundations didn't move, the top floor just got taller.