Picture an operations lead whose team built an assistant over the company's procedures and contracts. It answers some questions well and misses others that are plainly in the documents. The first instinct is a bigger model. The setting nobody looked at is how the documents were cut into pieces before indexing: every retrieval system splits documents into chunks, embeds each, and returns the closest few to a question, and the chunk size came from a library default. Five published evaluations, read together, say that default is a guess, and that the right value cannot be looked up, because it moved every time someone measured it on a different corpus.
What the default chunk size cost.
Chroma's report by Brandon Smith and Anton Troynikov measured chunking at the token level: did the exact passage that answers the question reach the model, and how much unrelated text came with it.[1] They built five small corpora (a State of the Union address, Wikitext, chat logs, ConvFinQA financial filings and PubMed articles, 328,208 tokens in total) and had GPT-4-Turbo write questions plus the verbatim excerpts that answer them, filtered for duplicates and weak matches, 472 in all. For each method they retrieved five chunks with OpenAI's text-embedding-3-large and scored recall (the share of answer tokens that came back) and precision (the share of returned tokens that were answer tokens).
The result that travels is about defaults. The report notes that OpenAI's documented default for its file search at the time was 800-token chunks with a 400-token overlap. Tested that way, a plain token splitter reached 87.9% recall but 1.4% precision: of every hundred tokens handed to the model, between one and two belonged to the answer. A recursive character splitter at 200 tokens with no overlap kept recall at 88.1% and raised precision to 7.0%. A chunker that asks an LLM where to cut reached 91.9% recall, the highest in the table, at 3.9% precision.[1] The authors' summary: "Notably, we find that default settings for certain popular chunking strategies can lead to relatively poor performance."
Two readings of fig 1 are safe. Overlap bought almost nothing: at 400 tokens, dropping it from 200 to zero moved recall from 88.1 to 89.5 and precision from 3.3 to 3.6, while halving the stored vectors. And shrinking the chunk raised precision far more than it cost recall, down to 200 tokens; the authors' own line is that the recursive splitter "with chunk size 200 and no overlap performs well", and the 400-token rows sit close behind. Their caveat is the one to keep: the dataset is small, the questions were model-written, and recall standard deviations run from about 25 to 37 points, so a point or two between rows means nothing. The abstract's "some strategies outperforming others by up to 9% in recall" names no pair; the widest gap in Table 1 is 8.3 points, between the Kamradt semantic chunker at 83.6 and the LLM chunker at 91.9.[1]
Semantic chunking rarely paid for itself.
"Semantic chunking" means cutting where the meaning changes rather than every N tokens: embed each sentence, start a new chunk when the next sentence is far from the last. It costs an embedding call per sentence at indexing time, and it is the method most often sold as the upgrade from fixed-size splitting. Renyi Qu, Ruixuan Tu and Forrest Sheng Bao tested that claim in "Is Semantic Chunking Worth the Computational Cost?", Findings of NAACL 2025, the one peer-reviewed source here.[2] Two authors work at Vectara, a retrieval vendor; the finding cuts against the expensive method.
They compared a fixed-size chunker (a set number of sentences per chunk) with two semantic chunkers, one cutting at a distance threshold between consecutive sentences and one clustering similar sentences, on document retrieval, evidence retrieval and answer generation. Where semantic chunking won, it won on documents the authors had stitched together from unrelated passages. They score document retrieval with F1@5, one number that combines how many of the five returned chunks were right and how many right ones were found. On stitched Miracl the breakpoint chunker reached 81.89% F1@5 against 69.45% for fixed-size; on stitched Natural Questions, 63.93% against 43.79%. On real documents the order reversed or vanished: HotpotQA 90.59% fixed-size against 87.37% and 84.79% for the two semantic methods, MSMARCO 93.58% against 92.23% and 93.18%.[2]
The authors' conclusion: "these benefits were highly context-dependent and did not consistently justify the additional computational cost", and "fixed-size chunking remains a more efficient and reliable choice for practical RAG applications". They name where it helped: "Semantic chunking occasionally improved performance, particularly on stitched datasets with high topic diversity."[2] That matches the Chroma table, where the threshold chunker landed at the bottom on recall. For an operator: the expensive method is not the safe default. If your documents really are a jumble of unrelated topics in one file, test it. Otherwise test the cheap methods first.
The best chunk size moved with the corpus.
The Fraunhofer IAIS preprint by Sinchana Ramakanth Bhat, Max Rudat, Jannis Spiekermann and Nicolas Flores-Herr asked the size question across six long-document question-answering datasets, two embedding models, and chunks of 64, 128, 256, 512 and 1,024 tokens with no overlap.[3] Their metric is Recall@1: did the single top chunk contain the answer. On SQuAD, where answers are short facts, the smallest chunk won: 0.642 at 64 tokens with the Stella model, falling to 0.386 at 1,024. On TechQA, technical support documents, the order inverted: 0.049 at 64 tokens, 0.614 at 512, 0.619 at 1,024, and 0.715 at 1,024 with the Snowflake model. On NarrativeQA, recall rose with size all the way up.
The abstract states it directly: "smaller chunks (64-128 tokens) are optimal for datasets with concise, fact-based answers", while "larger chunks (512-1024 tokens) improve retrieval in datasets requiring broader contextual understanding". The paper also finds "chunking effectiveness is model-dependent": on Natural Questions, Stella peaked at 512 tokens and Snowflake at 1,024.[3]
Two industry studies on business documents point the same way without agreeing on a number. Unstructured's preprint by Antonio Jimeno Yepes, Yao You, Jan Milczek, Sebastian Laverde and Leah Li tested 141 FinanceBench questions over 80 SEC filings averaging 147 pages.[4] Among plain token chunks, 256 tokens found the right page most often (73.05%, against 68.09% for 512), yet 512 gave the best answers (48.23% correct on manual grading, against 36.88% for 256). The paper, citing earlier work, puts it as "the best basic chunk size varies from data to data", and its own two tables bear that out: 256 wins on finding the page, 512 on answering. The authors' own method, cutting by document elements (titles, paragraphs, tables) found by a layout model, reached 84.40% page accuracy only when its three variants were pooled, with about half the chunks of the pooled token methods at 83.69%; its single variants scored below the token baselines on page accuracy and above them on answers (up to 53.19%). Unstructured sells that pipeline.
| source · corpus | sizes tested | what won | metric |
|---|---|---|---|
| Chroma [1] · 5 small mixed corpora | 200–800 tokens, overlap 0–400 | 200 tokens, no overlap (authors); 400-token rows close; LLM chunker highest recall | token-level recall and precision, 5 chunks |
| Fraunhofer [3] · SQuAD* | 64–1,024 tokens | 64 tokens (0.642) | Recall@1, Stella |
| Fraunhofer [3] · TechQA* | 64–1,024 tokens | 1,024 tokens (0.619; 0.715 with Snowflake) | Recall@1 |
| Unstructured [4] · 80 SEC filings | 128, 256, 512 tokens; element-based | 256 for page retrieval (73.05%); 512 for answers (48.23%); element-based pooled 84.40% | page accuracy; manual answer grading |
| NVIDIA [5] · FinanceBench | 128–2,048 tokens; page; section | 1,024 tokens (0.579), page-level close (0.566) | LLM-judged answer accuracy |
| NVIDIA [5] · Earnings | same | 512 tokens (0.681) | LLM-judged answer accuracy |
| NVIDIA [5] · five datasets pooled | same | page-level (0.648, std 0.107); token sizes 0.603–0.645 | LLM-judged answer accuracy |
NVIDIA's engineering post by Steve Han ran token chunks from 128 to 2,048 tokens at 15% overlap, plus page-level and section-level splitting, through its own retrieval stack on five document sets, scoring the final answers with two LLM judges.[5] Averaged over all five, page-level chunking scored 0.648 with the lowest spread (standard deviation 0.107), against 0.603 to 0.645 for the token sizes: "Page-level chunking is the overall winner." Per dataset the winner changed: 1,024 tokens on FinanceBench (0.579, page-level 0.566), 512 on the earnings reports (0.681), page-level on a long-form business set (0.520). The post's general line: "Very small (128 tokens) and very large (2,048 tokens) chunks generally underperformed medium-sized chunks." Its overlap test ran on one dataset at one size, and the post says it was "not a full-grid search".
Put the table together and no size wins across the board: 64 tokens on short-answer questions, 200 to 400 on Chroma's mixed set, 256 for finding the right page in a filing, 512 on earnings decks, 1,024 on support articles and FinanceBench, whole pages across NVIDIA's pool. The value depends on what is asked and of which documents, and the only place to read it is off your own corpus.
Read these results carefully.
The metrics differ, and they do not have to agree. Chroma scores tokens; Fraunhofer scores whether the top chunk had the answer; Unstructured scores pages and grades answers; NVIDIA grades only the final answer, with a model as judge. Unstructured shows why this matters: 256-token chunks found the right page most often, 512-token chunks produced the most correct answers. A chunk can be found and still carry too little context to answer from.
Several corpora are synthetic, and many questions are model-written. Fraunhofer stitched four of six datasets into long documents; the NAACL paper's clearest semantic-chunking wins came on stitched documents built to have abrupt topic changes. Procedures, contracts and quotes are not assembled that way, and on real documents fixed-size splitting held up. Chroma's 472 questions came from GPT-4-Turbo, at roughly one cent each by the authors' estimate (which they state for GPT-4); model-written questions tend to reuse the document's own phrasing, which flatters retrieval. Your users ask in their own words, which is why the golden set below starts from real questions.
Variance is large and sometimes unreported. Chroma's recall standard deviations are ±25 to ±37 points on means in the high 80s. NVIDIA reports one only for the pooled average. Fraunhofer reports none, and its text disagrees with its own table at least once (it says Natural Questions peaked at 1,024 tokens for both models while the table has Stella peaking at 512). Treat the direction of each result as the finding and the second decimal as noise. And four of the six sources sell something in the pipeline, so this note leans on the pattern across all of them rather than on any one table.
What chunking does not fix.
A chunk is embedded alone. "The clause above does not apply to orders under the minimum" means nothing without the clause above, and no chunk size solves that: make the chunk big enough to include it and precision drops, as fig 1 shows. One published remedy attacks this directly, and it is vendor work.
Anthropic's September 2024 post on contextual retrieval has a model write a short sentence of context for each chunk ("this chunk is from the 2023 supplier agreement with X, section on minimum order quantities") and prepends it before embedding. Across their test sets, the share of questions where the right chunk was missing from the top 20 fell from 5.7% to 3.7% with contextual embeddings, to 2.9% when a keyword index (BM25) was given the same context, and to 1.9% with a reranker on top: failed retrievals reduced "by 49% and, when combined with reranking, by 67%".[6] The one-time cost they report is $1.02 per million document tokens with prompt caching, on 800-token chunks. Those are Anthropic's numbers on Anthropic's datasets, not independently reproduced, and the gain is on a failure rate that was already small. The remedy sits on top of a chunk-size decision, not in place of it. The same post makes a point worth more than any chunking advice: under about 200,000 tokens of documents (around 500 pages, by their estimate), you may not need retrieval at all.
Two more things no chunk size fixes. Keyword matching, covered in our note on hybrid retrieval: a part number or a person's name is found by a keyword index, not by meaning, and Anthropic's 49% figure came from adding exactly that. And drift: the Fraunhofer and NVIDIA results say the best setting depends on the document type, so a corpus whose mix changes over months can move away from the setting you chose without anyone touching the configuration. None of the sources measured that over time; it follows from what they did measure, and the last step below is the answer.
Tune it in a day.
Acceleratech does not publish chunking benchmarks. We do set up retrieval systems for small and mid-size businesses, and this is the procedure we would run on one. The steps and the scale are ours; the numbers inside them are the sources'. One working day of one person who can run the pipeline, plus four to six re-indexings of the corpus, is the target for one corpus, not a measured duration. It needs no new model, vector database or semantic chunker.
- Build a golden set of about 50 real questions.
Pull them from what people actually asked: support tickets, the assistant's logs, the questions the owner answers by hand every week. For each, record the passage in your documents that answers it (the file and the exact text). Chroma's authors paid roughly a cent per question to have a model write theirs; that is the fallback if you have no real questions yet, knowing it flatters retrieval.[1]
→ the test your retrieval has to pass, in your users' words - Pick four to six settings, not thirty.
Fixed-size at three sizes (say 256, 512 and 1,024 tokens, the span where the token-based studies overlap), one of them also with a modest overlap; a page-level or heading-based split if your documents have real structure; semantic chunking only if your documents genuinely jump between unrelated topics in one file. Keep the embedding model fixed: Fraunhofer found the best size changes with it.[3]
→ a grid you can finish today - Measure recall at the k you actually use.
For each setting, index the corpus, run the 50 questions, and count how often the recorded passage appears in the top k chunks, k being what you pass to the model in production. Record tokens per retrieval too: that is the precision side, and in Chroma's table it varied five-fold at the same recall.[1] Then read the ten worst misses: is the problem the cut (the answer straddles two chunks), the words (a product code the embedding does not match) or the context (the chunk is right but unintelligible alone)?
→ recall@k and tokens per query, per setting, plus a list of named failures - Pick, and write down why.
Choose the smallest setting whose recall is within noise of the best; on 50 questions one question is two points, so two settings within two points are tied. Smaller chunks mean fewer tokens per answer. Record the setting, the date, the corpus size, the embedding model and the recall, next to the configuration in the repository.
→ a configuration with a measurement attached - If the misses are about keywords or context, fix those next.
Chunk size will not close them. Add a keyword index alongside the embeddings for codes and names. Consider contextual retrieval if the misses are "right document, wrong understanding", with Anthropic's $1.02 per million tokens as the published indexing cost and their 5.7% to 1.9% as a vendor's result.[6] Re-run the same 50 questions after each change.
→ the next lever, chosen from evidence rather than from the menu - Re-measure when the corpus changes.
Keep the golden set in the repository, run it on a schedule and whenever a new document type arrives in volume, and add questions as the corpus grows. Recall that falls when the configuration did not change is drift in the documents; repeat steps 2 to 4. Our note on eval harnesses covers keeping that set from rotting.
→ the setting stays right as the documents move
What this note does not claim.
No Acceleratech benchmark. We ran none of these numbers. An earlier version of this note built its case on an illustrative eight-strategy, five-corpus comparison, labelled as such under our editorial baseline. This revision replaces it with published evaluations: every benchmark figure here is from the six sources, as they report it; the question counts and settings in the how-to section are our suggested procedure, not measurements.
No recommended size. A reader who leaves with "use 400 tokens" has read this note wrong: that is what Chroma found on five small corpora with one embedding model, and Fraunhofer's TechQA result alone shows it does not generalize.
The one-day procedure is ours, not measured. Fifty questions and four to six settings are our judgment of what a small team can finish and still see a difference, not a sample-size calculation.
The context remedy is a vendor result. Anthropic's 67% reduction was measured by Anthropic, with its own model, on datasets it chose, and has not been independently reproduced in anything we read.
Drift is inferred, not measured. None of the sources tracked retrieval quality as a corpus changed over time. We say it follows from the finding that the best setting depends on the document mix; the re-measurement step is how to check that on your own system.
If an assistant over your documents misses answers that are plainly in them, describe the corpus and the kind of questions it gets (no documents, no customer data) through the contact form. We will write back, free, with the golden-set plan and the settings we would test first. Retrieval that is measured rather than assumed is part of our workflow automation services.