Semantic chunking and ingestion: splitting documents so retrieval works
Fixed-size chunking slices ideas in half. Semantic chunking splits where the meaning shifts — and injecting document structure into each chunk keeps it from being stranded. Here's the algorithm and the ingestion around it.
Chunking quietly sets the ceiling on a RAG system (the grounding post makes that case). The lazy default — split every N characters — is a blunt instrument that slices a thought in half. Semantic chunking is the technique that comes up when an interviewer asks you to 'show the architecture', so here's the algorithm, and the ingestion steps that surround it.
The semantic chunking algorithm
Instead of cutting at a character count, you cut where the topic shifts:
- Segment the document into sentences (spaCy or NLTK).
- Embed each sentence into a vector.
- Measure the cosine distance between each pair of adjacent sentences — a small distance means the topic is continuing, a spike means it has shifted.
- Detect boundaries with a dynamic threshold, not a fixed one: a rolling mean plus k·σ, or a percentile of the observed distances, so the cutoff adapts to the document.
- Assemble the sentences between boundaries into chunks.
The payoff: every chunk is a coherent, self-contained idea, so its embedding represents one topic instead of an arbitrary window that straddles two.
Parse the layout before you chunk
Don't treat a document as a flat string. Extract its structural metadata first — headers, tables, section hierarchy — so chunking respects real boundaries and a table isn't sliced down the middle (the Document-AI post covers layout-aware parsing). Layout is meaning, and throwing it away before you chunk throws away the structure that would have made your chunks good.
Inject metadata into each chunk
A chunk embedded in isolation loses its context — 'revenue grew 12%' is unanswerable without knowing whose and when. So prepend the document title, the section header, and the parent path into the chunk's text before you embed it. Now the embedding carries global context and the chunk is self-locating, which measurably lifts retrieval (it's the same idea as the contextual retrieval mentioned in the grounding post).
Index small, generate from big: parent-child
The complement to good chunking is decoupling what you match from what you return. Index small chunks for precise matching, but feed the larger parent block to the model for generation — so retrieval stays precise while the model still gets enough surrounding context to answer well (the advanced-RAG post goes deeper). Semantic chunking makes the small chunks coherent; parent-child makes sure coherence doesn't cost you context.
Fixed-size chunking asks 'where's the next 500 characters?'. Semantic chunking asks 'where does the idea end?' — and a chunk that holds one whole idea, tagged with where it came from, is one retrieval can actually use.