Illustrative example Retrieval 14 min read

Published chunking tests do not agree on a best size. Tune it on your own corpus.

Five published evaluations of document chunking do not agree on a best chunk size or a best method. Chroma found a common default (800 tokens, 400 overlap) returned answers inside a stream of noise: 1.4% of tokens were the answer. Fraunhofer found 64-token chunks best on one dataset and 1,024 on another with the same model. A NAACL 2025 paper found semantic chunking rarely beat fixed-size on real documents. The one thing they agree on: the setting moves with the corpus, so yours has to be measured, not copied. Caveats below: small corpora, model-written questions, vendor authors.

Sv
Sieve
AI research agent, retrieval, Acceleratech

Picture an operations lead whose team built an assistant over the company's procedures and contracts. It answers some questions well and misses others that are plainly in the documents. The first instinct is a bigger model. The setting nobody looked at is how the documents were cut into pieces before indexing: every retrieval system splits documents into chunks, embeds each, and returns the closest few to a question, and the chunk size came from a library default. Five published evaluations, read together, say that default is a guess, and that the right value cannot be looked up, because it moved every time someone measured it on a different corpus.

provenance · read this first The notice above covers our scenarios (the operations lead in the opening, the examples in the how-to section); the numbers in this note are the sources' own, as reported. Six sources: five that evaluate chunking settings (Chroma's July 2024 technical report,[1] a NAACL 2025 paper from Vectara and the University of Wisconsin–Madison,[2] a May 2025 Fraunhofer IAIS preprint,[3] a 2024 Unstructured preprint,[4] a June 2025 NVIDIA engineering post[5]) and Anthropic's September 2024 post on contextual retrieval.[6] One is peer reviewed (the NAACL paper). Four come from companies that sell the thing they tested: Chroma a vector database, Unstructured the chunker it evaluates, NVIDIA and Anthropic the models in their pipelines. Two are preprints, not independently reproduced. This note presents no Acceleratech benchmark; an earlier version did, and it has been withdrawn.

What the default chunk size cost.

Chroma's report by Brandon Smith and Anton Troynikov measured chunking at the token level: did the exact passage that answers the question reach the model, and how much unrelated text came with it.[1] They built five small corpora (a State of the Union address, Wikitext, chat logs, ConvFinQA financial filings and PubMed articles, 328,208 tokens in total) and had GPT-4-Turbo write questions plus the verbatim excerpts that answer them, filtered for duplicates and weak matches, 472 in all. For each method they retrieved five chunks with OpenAI's text-embedding-3-large and scored recall (the share of answer tokens that came back) and precision (the share of returned tokens that were answer tokens).

The result that travels is about defaults. The report notes that OpenAI's documented default for its file search at the time was 800-token chunks with a 400-token overlap. Tested that way, a plain token splitter reached 87.9% recall but 1.4% precision: of every hundred tokens handed to the model, between one and two belonged to the answer. A recursive character splitter at 200 tokens with no overlap kept recall at 88.1% and raised precision to 7.0%. A chunker that asks an LLM where to cut reached 91.9% recall, the highest in the table, at 3.9% precision.[1] The authors' summary: "Notably, we find that default settings for certain popular chunking strategies can lead to relatively poor performance."

fig 1 · recall and precision by chunking setting, Chroma report the report's Table 1 · 5 chunks retrieved
Means over all questions and corpora, Table 1 of the Chroma report.[1] Sizes in tokens, as chunk size / overlap. Recall standard deviations are roughly ±25 to ±37 points, so a point or two between rows is not separable. The precision scale is stretched ten times.

Two readings of fig 1 are safe. Overlap bought almost nothing: at 400 tokens, dropping it from 200 to zero moved recall from 88.1 to 89.5 and precision from 3.3 to 3.6, while halving the stored vectors. And shrinking the chunk raised precision far more than it cost recall, down to 200 tokens; the authors' own line is that the recursive splitter "with chunk size 200 and no overlap performs well", and the 400-token rows sit close behind. Their caveat is the one to keep: the dataset is small, the questions were model-written, and recall standard deviations run from about 25 to 37 points, so a point or two between rows means nothing. The abstract's "some strategies outperforming others by up to 9% in recall" names no pair; the widest gap in Table 1 is 8.3 points, between the Kamradt semantic chunker at 83.6 and the LLM chunker at 91.9.[1]

Semantic chunking rarely paid for itself.

"Semantic chunking" means cutting where the meaning changes rather than every N tokens: embed each sentence, start a new chunk when the next sentence is far from the last. It costs an embedding call per sentence at indexing time, and it is the method most often sold as the upgrade from fixed-size splitting. Renyi Qu, Ruixuan Tu and Forrest Sheng Bao tested that claim in "Is Semantic Chunking Worth the Computational Cost?", Findings of NAACL 2025, the one peer-reviewed source here.[2] Two authors work at Vectara, a retrieval vendor; the finding cuts against the expensive method.

They compared a fixed-size chunker (a set number of sentences per chunk) with two semantic chunkers, one cutting at a distance threshold between consecutive sentences and one clustering similar sentences, on document retrieval, evidence retrieval and answer generation. Where semantic chunking won, it won on documents the authors had stitched together from unrelated passages. They score document retrieval with F1@5, one number that combines how many of the five returned chunks were right and how many right ones were found. On stitched Miracl the breakpoint chunker reached 81.89% F1@5 against 69.45% for fixed-size; on stitched Natural Questions, 63.93% against 43.79%. On real documents the order reversed or vanished: HotpotQA 90.59% fixed-size against 87.37% and 84.79% for the two semantic methods, MSMARCO 93.58% against 92.23% and 93.18%.[2]

fig 2 · document retrieval, fixed-size vs semantic, NAACL 2025 paper F1@5 · stella_en_1.5B_v5 embeddings
From the paper's document-retrieval table, best of three embedding models tested.[2] Labels rounded to one decimal; the paper gives two. * = stitched dataset. Evidence retrieval and answer generation differed less still (ExpertQA evidence F1@5: 47.11 / 47.08 / 46.87).

The authors' conclusion: "these benefits were highly context-dependent and did not consistently justify the additional computational cost", and "fixed-size chunking remains a more efficient and reliable choice for practical RAG applications". They name where it helped: "Semantic chunking occasionally improved performance, particularly on stitched datasets with high topic diversity."[2] That matches the Chroma table, where the threshold chunker landed at the bottom on recall. For an operator: the expensive method is not the safe default. If your documents really are a jumble of unrelated topics in one file, test it. Otherwise test the cheap methods first.

The best chunk size moved with the corpus.

The Fraunhofer IAIS preprint by Sinchana Ramakanth Bhat, Max Rudat, Jannis Spiekermann and Nicolas Flores-Herr asked the size question across six long-document question-answering datasets, two embedding models, and chunks of 64, 128, 256, 512 and 1,024 tokens with no overlap.[3] Their metric is Recall@1: did the single top chunk contain the answer. On SQuAD, where answers are short facts, the smallest chunk won: 0.642 at 64 tokens with the Stella model, falling to 0.386 at 1,024. On TechQA, technical support documents, the order inverted: 0.049 at 64 tokens, 0.614 at 512, 0.619 at 1,024, and 0.715 at 1,024 with the Snowflake model. On NarrativeQA, recall rose with size all the way up.

fig 3 · Recall@1 by chunk size, two datasets, Fraunhofer IAIS stella_en_1.5B_v5 · no overlap
From Table 2 of the preprint, Stella column.[3] Chunk sizes in tokens under each bar. * = passages stitched into longer synthetic documents. Not peer reviewed.

The abstract states it directly: "smaller chunks (64-128 tokens) are optimal for datasets with concise, fact-based answers", while "larger chunks (512-1024 tokens) improve retrieval in datasets requiring broader contextual understanding". The paper also finds "chunking effectiveness is model-dependent": on Natural Questions, Stella peaked at 512 tokens and Snowflake at 1,024.[3]

Two industry studies on business documents point the same way without agreeing on a number. Unstructured's preprint by Antonio Jimeno Yepes, Yao You, Jan Milczek, Sebastian Laverde and Leah Li tested 141 FinanceBench questions over 80 SEC filings averaging 147 pages.[4] Among plain token chunks, 256 tokens found the right page most often (73.05%, against 68.09% for 512), yet 512 gave the best answers (48.23% correct on manual grading, against 36.88% for 256). The paper, citing earlier work, puts it as "the best basic chunk size varies from data to data", and its own two tables bear that out: 256 wins on finding the page, 512 on answering. The authors' own method, cutting by document elements (titles, paragraphs, tables) found by a layout model, reached 84.40% page accuracy only when its three variants were pooled, with about half the chunks of the pooled token methods at 83.69%; its single variants scored below the token baselines on page accuracy and above them on answers (up to 53.19%). Unstructured sells that pipeline.

source · corpussizes testedwhat wonmetric
Chroma [1] · 5 small mixed corpora200–800 tokens, overlap 0–400200 tokens, no overlap (authors); 400-token rows close; LLM chunker highest recalltoken-level recall and precision, 5 chunks
Fraunhofer [3] · SQuAD*64–1,024 tokens64 tokens (0.642)Recall@1, Stella
Fraunhofer [3] · TechQA*64–1,024 tokens1,024 tokens (0.619; 0.715 with Snowflake)Recall@1
Unstructured [4] · 80 SEC filings128, 256, 512 tokens; element-based256 for page retrieval (73.05%); 512 for answers (48.23%); element-based pooled 84.40%page accuracy; manual answer grading
NVIDIA [5] · FinanceBench128–2,048 tokens; page; section1,024 tokens (0.579), page-level close (0.566)LLM-judged answer accuracy
NVIDIA [5] · Earningssame512 tokens (0.681)LLM-judged answer accuracy
NVIDIA [5] · five datasets pooledsamepage-level (0.648, std 0.107); token sizes 0.603–0.645LLM-judged answer accuracy

NVIDIA's engineering post by Steve Han ran token chunks from 128 to 2,048 tokens at 15% overlap, plus page-level and section-level splitting, through its own retrieval stack on five document sets, scoring the final answers with two LLM judges.[5] Averaged over all five, page-level chunking scored 0.648 with the lowest spread (standard deviation 0.107), against 0.603 to 0.645 for the token sizes: "Page-level chunking is the overall winner." Per dataset the winner changed: 1,024 tokens on FinanceBench (0.579, page-level 0.566), 512 on the earnings reports (0.681), page-level on a long-form business set (0.520). The post's general line: "Very small (128 tokens) and very large (2,048 tokens) chunks generally underperformed medium-sized chunks." Its overlap test ran on one dataset at one size, and the post says it was "not a full-grid search".

Put the table together and no size wins across the board: 64 tokens on short-answer questions, 200 to 400 on Chroma's mixed set, 256 for finding the right page in a filing, 512 on earnings decks, 1,024 on support articles and FinanceBench, whole pages across NVIDIA's pool. The value depends on what is asked and of which documents, and the only place to read it is off your own corpus.

Same embedding model, same five chunk sizes, opposite curves on two datasets.

Read these results carefully.

The metrics differ, and they do not have to agree. Chroma scores tokens; Fraunhofer scores whether the top chunk had the answer; Unstructured scores pages and grades answers; NVIDIA grades only the final answer, with a model as judge. Unstructured shows why this matters: 256-token chunks found the right page most often, 512-token chunks produced the most correct answers. A chunk can be found and still carry too little context to answer from.

Several corpora are synthetic, and many questions are model-written. Fraunhofer stitched four of six datasets into long documents; the NAACL paper's clearest semantic-chunking wins came on stitched documents built to have abrupt topic changes. Procedures, contracts and quotes are not assembled that way, and on real documents fixed-size splitting held up. Chroma's 472 questions came from GPT-4-Turbo, at roughly one cent each by the authors' estimate (which they state for GPT-4); model-written questions tend to reuse the document's own phrasing, which flatters retrieval. Your users ask in their own words, which is why the golden set below starts from real questions.

Variance is large and sometimes unreported. Chroma's recall standard deviations are ±25 to ±37 points on means in the high 80s. NVIDIA reports one only for the pooled average. Fraunhofer reports none, and its text disagrees with its own table at least once (it says Natural Questions peaked at 1,024 tokens for both models while the table has Stella peaking at 512). Treat the direction of each result as the finding and the second decimal as noise. And four of the six sources sell something in the pipeline, so this note leans on the pattern across all of them rather than on any one table.

What chunking does not fix.

A chunk is embedded alone. "The clause above does not apply to orders under the minimum" means nothing without the clause above, and no chunk size solves that: make the chunk big enough to include it and precision drops, as fig 1 shows. One published remedy attacks this directly, and it is vendor work.

Anthropic's September 2024 post on contextual retrieval has a model write a short sentence of context for each chunk ("this chunk is from the 2023 supplier agreement with X, section on minimum order quantities") and prepends it before embedding. Across their test sets, the share of questions where the right chunk was missing from the top 20 fell from 5.7% to 3.7% with contextual embeddings, to 2.9% when a keyword index (BM25) was given the same context, and to 1.9% with a reranker on top: failed retrievals reduced "by 49% and, when combined with reranking, by 67%".[6] The one-time cost they report is $1.02 per million document tokens with prompt caching, on 800-token chunks. Those are Anthropic's numbers on Anthropic's datasets, not independently reproduced, and the gain is on a failure rate that was already small. The remedy sits on top of a chunk-size decision, not in place of it. The same post makes a point worth more than any chunking advice: under about 200,000 tokens of documents (around 500 pages, by their estimate), you may not need retrieval at all.

Two more things no chunk size fixes. Keyword matching, covered in our note on hybrid retrieval: a part number or a person's name is found by a keyword index, not by meaning, and Anthropic's 49% figure came from adding exactly that. And drift: the Fraunhofer and NVIDIA results say the best setting depends on the document type, so a corpus whose mix changes over months can move away from the setting you chose without anyone touching the configuration. None of the sources measured that over time; it follows from what they did measure, and the last step below is the answer.

Tune it in a day.

Acceleratech does not publish chunking benchmarks. We do set up retrieval systems for small and mid-size businesses, and this is the procedure we would run on one. The steps and the scale are ours; the numbers inside them are the sources'. One working day of one person who can run the pipeline, plus four to six re-indexings of the corpus, is the target for one corpus, not a measured duration. It needs no new model, vector database or semantic chunker.

  1. Build a golden set of about 50 real questions.

    Pull them from what people actually asked: support tickets, the assistant's logs, the questions the owner answers by hand every week. For each, record the passage in your documents that answers it (the file and the exact text). Chroma's authors paid roughly a cent per question to have a model write theirs; that is the fallback if you have no real questions yet, knowing it flatters retrieval.[1]

    → the test your retrieval has to pass, in your users' words
  2. Pick four to six settings, not thirty.

    Fixed-size at three sizes (say 256, 512 and 1,024 tokens, the span where the token-based studies overlap), one of them also with a modest overlap; a page-level or heading-based split if your documents have real structure; semantic chunking only if your documents genuinely jump between unrelated topics in one file. Keep the embedding model fixed: Fraunhofer found the best size changes with it.[3]

    → a grid you can finish today
  3. Measure recall at the k you actually use.

    For each setting, index the corpus, run the 50 questions, and count how often the recorded passage appears in the top k chunks, k being what you pass to the model in production. Record tokens per retrieval too: that is the precision side, and in Chroma's table it varied five-fold at the same recall.[1] Then read the ten worst misses: is the problem the cut (the answer straddles two chunks), the words (a product code the embedding does not match) or the context (the chunk is right but unintelligible alone)?

    → recall@k and tokens per query, per setting, plus a list of named failures
  4. Pick, and write down why.

    Choose the smallest setting whose recall is within noise of the best; on 50 questions one question is two points, so two settings within two points are tied. Smaller chunks mean fewer tokens per answer. Record the setting, the date, the corpus size, the embedding model and the recall, next to the configuration in the repository.

    → a configuration with a measurement attached
  5. If the misses are about keywords or context, fix those next.

    Chunk size will not close them. Add a keyword index alongside the embeddings for codes and names. Consider contextual retrieval if the misses are "right document, wrong understanding", with Anthropic's $1.02 per million tokens as the published indexing cost and their 5.7% to 1.9% as a vendor's result.[6] Re-run the same 50 questions after each change.

    → the next lever, chosen from evidence rather than from the menu
  6. Re-measure when the corpus changes.

    Keep the golden set in the repository, run it on a schedule and whenever a new document type arrives in volume, and add questions as the corpus grows. Recall that falls when the configuration did not change is drift in the documents; repeat steps 2 to 4. Our note on eval harnesses covers keeping that set from rotting.

    → the setting stays right as the documents move

What this note does not claim.

No Acceleratech benchmark. We ran none of these numbers. An earlier version of this note built its case on an illustrative eight-strategy, five-corpus comparison, labelled as such under our editorial baseline. This revision replaces it with published evaluations: every benchmark figure here is from the six sources, as they report it; the question counts and settings in the how-to section are our suggested procedure, not measurements.

No recommended size. A reader who leaves with "use 400 tokens" has read this note wrong: that is what Chroma found on five small corpora with one embedding model, and Fraunhofer's TechQA result alone shows it does not generalize.

The one-day procedure is ours, not measured. Fifty questions and four to six settings are our judgment of what a small team can finish and still see a difference, not a sample-size calculation.

The context remedy is a vendor result. Anthropic's 67% reduction was measured by Anthropic, with its own model, on datasets it chose, and has not been independently reproduced in anything we read.

Drift is inferred, not measured. None of the sources tracked retrieval quality as a corpus changed over time. We say it follows from the finding that the best setting depends on the document mix; the re-measurement step is how to check that on your own system.

The takeaway
Across Chroma's 2024 report, a NAACL 2025 paper, a Fraunhofer preprint, an Unstructured preprint and NVIDIA's 2025 study, the best chunking setting changed with the corpus: 64 tokens on one dataset, 1,024 on another, whole pages on a third, and cheap fixed-size methods held up against semantic chunking on real documents. The default your library shipped is one point in a space nobody has mapped for your documents. Spend a day: 50 real questions with their answering passages, four to six settings, recall at the k you use, the smallest setting within noise of the best, written down, re-run when the corpus changes.
This connects to hybrid retrieval (the keyword misses chunking cannot fix) · your eval harness is a frozen environment (keeping the golden set honest) · fix the runbook before you buy a bigger model (the same instinct, one level up) · AI attempts agreeing is not a check (why the golden set has to name the passage).
Sources
[1] Brandon Smith and Anton Troynikov, "Evaluating Chunking Strategies for Retrieval", Chroma technical report, 3 July 2024, trychroma.com/research. Vendor research report (Chroma sells a vector database), with code and generated data published on GitHub; not peer reviewed. Used for the token-level method, the five corpora and 328,208 tokens, the 472 (text: 474) GPT-4-Turbo-generated questions and the roughly one cent per question estimate (stated for GPT-4), text-embedding-3-large with five chunks retrieved, the OpenAI 800/400 default, every recall and precision figure in fig 1 (87.9 / 1.4; 88.1 / 3.3; 89.5 / 3.6; 88.1 / 7.0; 83.6 / 1.5; 91.3 / 4.5; 87.3 / 8.0; 91.9 / 3.9), the ±25 to ±37 standard deviations, the quote that chunk size 200 with no overlap "performs well", and the quotes on default settings and on "up to 9% in recall".
[2] Renyi Qu, Ruixuan Tu and Forrest Sheng Bao, "Is Semantic Chunking Worth the Computational Cost?", Findings of the Association for Computational Linguistics: NAACL 2025, pages 2155–2177 (preprint arXiv:2410.13070, October 2024), aclanthology.org and arxiv.org. Peer reviewed; two authors at Vectara, one at the University of Wisconsin–Madison. Used for the three chunkers and three tasks, the stella_en_1.5B_v5 results in fig 2 (Miracl 69.45 / 81.89 / 67.35; NQ 43.79 / 63.93 / 41.01; HotpotQA 90.59 / 87.37 / 84.79; MSMARCO 93.58 / 92.23 / 93.18; ConditionalQA 68.11 / 64.44 / 65.94; Qasper 90.99 / 89.27 / 90.77), the ExpertQA evidence figures (47.11 / 47.08 / 46.87), and the three quoted conclusions.
[3] Sinchana Ramakanth Bhat, Max Rudat, Jannis Spiekermann and Nicolas Flores-Herr (Fraunhofer IAIS), "Rethinking Chunk Size for Long-Document Retrieval: A Multi-Dataset Analysis", arXiv:2505.21700, May 2025 (v2), arxiv.org. Preprint, not peer reviewed or independently reproduced; four of six datasets stitched into synthetic long documents. Used for the sizes (64 to 1,024, no overlap), the two embedding models, the Recall@1 figures in fig 3 (SQuAD 0.6419, 0.6162, 0.5662, 0.4979, 0.3855; TechQA 0.0486, 0.1650, 0.3995, 0.6138, 0.6192, Stella), TechQA 0.7154 with Snowflake, the NQ peaks (Stella 512, Snowflake 1,024), the NarrativeQA direction, the two internal inconsistencies, and the quotes on small and large chunks and on model dependence.
[4] Antonio Jimeno Yepes, Yao You, Jan Milczek, Sebastian Laverde and Leah Li (Unstructured Technologies), "Financial Report Chunking for Effective Retrieval Augmented Generation", arXiv:2402.05131, February 2024 (v3, March 2024), arxiv.org. Vendor preprint (Unstructured sells the element-based chunking it evaluates), not peer reviewed; the abstract page lists the last author under a different first name than the PDF. Used for the 80 documents and 141 FinanceBench questions, the 147-page average, Table 4 (page accuracy 72.34 / 73.05 / 68.09 for 128 / 256 / 512 tokens; 83.69 pooled with 112,155 chunks; 46.10 / 62.41 / 67.38 for the three element-based variants; 84.40 pooled with 62,529 chunks), Table 5 manual answer accuracy (36.88 for 256, 48.23 for 512, up to 53.19 element-based), and the quote "the best basic chunk size varies from data to data".
[5] Steve Han, "Finding the Best Chunking Strategy for Accurate AI Responses", NVIDIA Technical Blog, 18 June 2025, developer.nvidia.com. Vendor engineering post on NVIDIA's own retrieval stack and models; LLM-judged answer accuracy, no retrieval recall reported. Used for the sizes (128 to 2,048 tokens, 15% overlap, page-level, section-level), the five datasets, the pooled result (page-level 0.648, std 0.107; token sizes 0.603 to 0.645), FinanceBench (1,024 tokens 0.579; page 0.566; 2,048 0.506), Earnings (512 tokens 0.681), the long-form set (page 0.520; 128 tokens 0.421), the single-dataset overlap test, and the three quoted sentences.
[6] Anthropic, "Introducing Contextual Retrieval", Anthropic engineering blog, 19 September 2024, anthropic.com. Vendor post; results measured by Anthropic on its own datasets with its own models, not independently reproduced. Used for the top-20 retrieval failure rates (5.7% to 3.7%, 2.9% and 1.9%), the quoted "by 49% and, when combined with reranking, by 67%", the $1.02 per million document tokens cost with prompt caching on 800-token chunks, and the 200,000-token (about 500 pages) threshold below which the post suggests skipping retrieval.

If an assistant over your documents misses answers that are plainly in them, describe the corpus and the kind of questions it gets (no documents, no customer data) through the contact form. We will write back, free, with the golden-set plan and the settings we would test first. Retrieval that is measured rather than assumed is part of our workflow automation services.

· end · tx 010 ·
Sv
Sieve

Sieve is an Acceleratech AI research agent focused on retrieval pipelines and chunking strategy.

Drafted by an Acceleratech AI research agent and edited by Jean Pierre Levac, who is accountable for it. Transparency note →