How do you choose a chunking strategy for a document corpus?

Start from the questions, not the file size

The chunk is the unit that gets embedded, scored and returned, so its size sets the smallest thing your system can find and the largest thing it can hand to the model in one slot. That makes chunking a retrieval decision. The useful starting point is a handful of real questions the corpus has to answer, and the observation of how much text it takes to answer each of them.

If the answers are single facts sitting in one sentence — a price, an entitlement, a retention period — small chunks win, because a short chunk has a focused embedding and a long one averages several topics into a vector that is close to none of them. If the answers are procedures or arguments that only make sense as a whole, small chunks are a liability, because a retriever that returns step four of seven has returned something worse than nothing. Most corpora contain both, which is why a single global size is usually a compromise rather than a choice.

The token limit of your generation call is a constraint on how many chunks you can pass, not a target for how large each should be. Sizing chunks to fill the window is the reasoning that produces 2,000-token blobs whose embeddings retrieve poorly and whose content buries the relevant sentence among four paragraphs the model has to ignore.

Structure first, token count as a cap

The strongest default is to split on the document's own boundaries — heading levels, list items, table rows, clause numbers in a contract, cells in a notebook — and to treat a maximum token count only as a backstop for a section that turns out to be enormous. This works because the author already grouped related material, and because those boundaries give you something a fixed offset cannot: a name for the chunk. A chunk that knows it is section 4.2 of the returns policy can carry that in its embedded text and in its citation.

Fixed-size splitting is defensible for genuinely unstructured text — chat transcripts, scanned prose with no reliable heading detection — and for a first baseline you intend to beat. It is not defensible for HTML, Markdown, spreadsheets or slide decks, where the structure is available and throwing it away is a choice.

Overlap is the cheap insurance for boundaries you got slightly wrong: a few sentences shared between neighbours means a fact that straddles a split appears whole in at least one chunk. It costs index size and creates near-duplicate results, so it is worth paying for where boundaries are guesses and worth skipping where they are structural. Overlapping table rows buys nothing.

Where a boundary manufactures a confident wrong answer

The failure everybody expects from bad chunking is a missing answer, and that failure is benign because you can see it. The dangerous one is the boundary that leaves a fragment that reads as complete and is not.

A table split mid-row is the clearest case. Retrieve Priority | 2 days without the header row and you have a value with no idea which column it came from, and a model asked about delivery times will attach it to whichever heading appears nearby. A clause severed from its qualifier is the same shape and harder to spot: "the deposit is refundable" is a sentence, and the paragraph that continued "except where cancellation occurs within 14 days" is in the next chunk. Nothing in the retrieved text signals that it has been truncated, so the generator produces a fluent, cited, wrong answer, and the citation makes it more believable rather than less.

This is why the mitigations are specific rather than general. Repeat the header row into every chunk of a long table. Keep a clause and its exceptions together even when that breaks your size cap, because the cap is a preference and the clause is the meaning. Prefer to overflow a chunk than to split a sentence, and never split inside a code block or a table row.

Making the choice measurable

Chunking arguments are unwinnable in the abstract, so turn the parameter into an experiment. Take fifty real questions with a known answer string, index the corpus under two strategies, and measure how often the answer string appears in the top k retrieved chunks. That number needs no model to grade and settles the argument in an afternoon. A useful second measurement is how many retrieved chunks the answer needed, because a strategy with equal recall but fewer chunks per answer leaves more of the window for everything else.

Two refinements are worth knowing about because they dissolve the size trade-off rather than splitting it. You can embed a small, focused chunk for retrieval but pass the model its surrounding parent section once it is selected, so matching stays precise while the context stays whole. And you can prefix each chunk's embedded text with its document title and section path, which fixes the orphaned-fragment case at the point where it actually hurts — in the vector, where the fragment otherwise matches nothing.

Chunking is not formatting. Every boundary you draw is a decision about which questions the corpus can still answer, and the boundaries that cost you most are the ones that leave behind a fragment that still reads like a sentence.