from evals import tables, report as R
from evals.queries import build as build_queries, FLAVOURS, overlap
print(tables._corpus())configuration tiers
The litesearch configuration a RAG pipeline should use for each kind of document, from measurements
What this measures
litesearch offers many choices: four chunk sizes, five encoders, late chunking, a document tree, an entity graph, clustering, a reranker, and two ways to prepare a keyword query. This notebook reports which combination works best for each of three kinds of document, and groups the results into configuration tiers, from a keyword-only setup with no model up to the most expensive one.
The measurements come from python -m evals.run, in evals/. It builds 45 stores and takes hours, so the harness lives there and this notebook only prints its tables. The full report and the tier definitions are in docs/rag_tiers.md. The cells are eval: false and have no saved output; run them from the repository root after the harness has run.
Several defaults have changed since this was measured, and partly because of it. chunk_markdown now cuts at 512 bytes, db.search now sends its keyword query through pre(), late chunking was deleted (its code is in evals/latechunk.py), and the graph leg was removed from context. Where the sections below describe a default, they describe the one in place at the time.
The three genres were chosen because each breaks retrieval in a different way:
| genre | what it is | what makes it hard |
|---|---|---|
| regulatory | 8 EU directives and regulations, 489 pages | real hierarchy, and the same terms repeated across documents |
| arxiv | 12 papers from cs and astro-ph | headings do not survive PDF conversion, and mathematical notation does not survive at all |
| astrology | 7 books printed between 1822 and 1920, 1,275 pages | archaic prose, CHAPTER XI in the text instead of markup, and one book with no headings |
The astro-ph papers are there on purpose. They talk about Mars, conjunctions and ascending nodes, but not in the way the astrology books do, so they test whether a retriever tells the genres apart or only matches words.
Ground truth without an LLM
Each query starts from a sentence that occurs exactly once in the corpus, so the passage and the section it belongs to are known. The sentence is then reworded in five flavours of increasing difficulty: verbatim, degraded, keyword, paraphrase and kw_para. All five are built from the same source sentences, so every comparison here is paired: two configurations are always scored on the same questions.
A result is scored by word 5-grams, runs of five words, that occur in only one section. That works at three levels (passage, section and document), and it does not change when the chunk size does, which matters because chunk size turns out to be the variable with the largest effect. Most tables report section MRR: the mean over queries of 1 divided by the rank of the first result from the right section.
qs = build_queries('astrology')
for fl in FLAVOURS: print(f'{fl:<11} {qs[3][fl]}')1. Chunk size
The four settings are page (one chunk per page) and chunks of at most 1,024, 512 and 256 characters, cut by chonkie’s RecursiveChunker. This was the largest effect in the whole evaluation. When it ran, the default chunker was FastChunker(chunk_size=4096) and add_doc fed it one section at a time, so on pages of 1,700 to 3,500 characters it almost never split anything: the default behaved like page. The default is now 512, counted in UTF-8 bytes: 512 characters of English, about 170 of Devanagari.
print(R.summary('grain', ['genre','chunking','encoder','strategy'], 'u_mrr', label='section MRR'))2. Encoder
An encoder turns text into an embedding, a vector of numbers placed so that texts with similar meaning sit close together. The five encoders tested, potion-32M, bge-small, jina-v2-sm, egemma-300m and nomic-v1.5, span three orders of magnitude of cost per chunk. All produce float16 vectors and all ran on the same 4 CPU cores.
print(R.summary('encoder', ['genre','encoder','strategy'], 'u_mrr', label='section MRR'))3. Retrieval strategy
A strategy is how the keyword side and the vector side of a search are run and combined. When this was measured, db.search quoted each token of the query, which made its keyword query match only rows holding every token. pre() turns a query into keywords with wildcards joined by OR, and the caller had to know to apply it. db.search now applies it by default (fts_pre=True).
print(R.summary('strategy', ['genre','encoder','strategy'], 'u_mrr', label='section MRR'))4. Structure
A flat store, the document tree, and the tree rebuilt with headings recovered from the PDF, since PDF conversion loses most of them.
print(R.summary('structure', ['genre','mode','encoder','strategy'], 'u_mrr', label='section MRR'))5. Late chunking
Late chunking embeds a whole document with a long-context encoder and then splits the result into chunk vectors, so each chunk’s vector carries some of its surroundings. It measured worse than embedding chunks on their own and has since been deleted.
print(R.summary('late', ['genre','mode','strategy'], 'u_mrr', label='section MRR'))6. The graph leg
A third ranked list, built by walking a graph of the entities that chunks mention, fused with the keyword and vector lists. It measured negative and is no longer part of context; it lives on in vruksha.
print(R.summary('graph', ['genre','strategy','graph_w'], 'u_mrr', label='section MRR'))
print(R.summary('graph_notopic', ['genre','mode','strategy'], 'u_mrr', label='same graph, topic nodes removed'))7. Clustering
Clustering is judged here as a map of the corpus, by whether its groups follow the genres and sections, not by whether it improves ranking.
print(tables._cluster())8. One store or three
All three genres in one store against one store per genre. The second table reports how many of the top 10 hits come from the wrong genre.
print(R.summary('mixed', ['genre','mode','encoder','strategy'], 'u_mrr', label='section MRR'))
print(R.summary('mixed', ['genre','mode','encoder','strategy'], 'cross_genre', label='share of top-10 hits from the wrong genre'))Cost
Indexing time and query latency for each configuration.
print(R.cost_table())The tiers
docs/rag_tiers.md has the tier definitions, the changes for each genre, and the list of features that did not pay for themselves.