from evals import tables, report as R
from evals.queries import build as build_queries, FLAVOURS, overlap
print(tables._corpus())configuration tiers
Which litesearch configuration should a RAG pipeline use, per document genre?
What this measures
litesearch has accumulated choices: four chunk granularities, five encoders, late chunking, a document tree, an entity graph, clustering, a reranker, and two ways to preprocess an FTS query. Nothing in the library says which combination to use, and the defaults are not the best combination for any of the three genres tested here.
This notebook reads the results of python -m evals.run and turns them into configuration tiers. The harness lives in evals/ rather than in this notebook, because it builds ~50 stores and takes hours, but every table below is generated from its output, and the full report with the tier definitions is in docs/rag_tiers.md.
Three genres, chosen because they fail differently:
| genre | what it is | how it breaks things |
|---|---|---|
| regulatory | 8 EU directives and regulations, 489 pages | real hierarchy, and enormous term repetition between documents |
| arxiv | 12 papers, cs.* and astro-ph | headings do not survive PDF conversion; notation does not survive at all |
| astrology | 7 books printed 1822–1920, 1,275 pages | archaic prose, CHAPTER XI instead of markup, one book with no headings |
The astro-ph papers are in there on purpose: they talk about Mars, conjunctions and ascending nodes without meaning any of it the way the astrology books do, which is the cheapest available test of whether a retriever separates genres or only matches words.
Ground truth without an LLM
Every query comes from one sentence that occurs exactly once in the corpus, so its passage and its section are known exactly. Five flavours degrade the wording progressively, and all five are asked of the same source sentences, so every comparison in this notebook is paired.
Scoring is keyed on word 5-grams that are unique to one section, at three levels, passage, section, document. Nothing in the metric moves when the chunk size does, which matters because chunk size is the variable with the largest effect.
qs = build_queries('astrology')
for fl in FLAVOURS: print(f'{fl:<11} {qs[3][fl]}')1. Chunk granularity
The single largest effect in the whole evaluation, and the one place where litesearch’s default is actively wrong: chunk_markdown uses FastChunker(chunk_size=4096) and add_doc feeds it one node segment at a time, so on pages of 1.7–3.5k characters the default chunker almost never splits anything. page below is the default.
print(R.summary('grain', ['genre','chunking','encoder','strategy'], 'u_mrr', label='section MRR'))2. Encoder
Five encoders spanning three orders of magnitude of cost per chunk, all normalised to float16 and all measured on the same 4 CPU cores.
print(R.summary('encoder', ['genre','encoder','strategy'], 'u_mrr', label='section MRR'))3. Retrieval strategy
db.search quotes each token of the query, which makes the FTS leg an implicit AND over every token. pre() exists in the library to turn a query into keywords with wildcards and OR, but search does not call it, and the caller has to know to.
print(R.summary('strategy', ['genre','encoder','strategy'], 'u_mrr', label='section MRR'))4. Structure: flat, tree, and the headings PDF conversion throws away
print(R.summary('structure', ['genre','mode','encoder','strategy'], 'u_mrr', label='section MRR'))5. Late chunking
print(R.summary('late', ['genre','mode','strategy'], 'u_mrr', label='section MRR'))6. The graph leg
print(R.summary('graph', ['genre','strategy','graph_w'], 'u_mrr', label='section MRR'))
print(R.summary('graph_notopic', ['genre','mode','strategy'], 'u_mrr', label='same graph, topic nodes removed'))7. Clustering, measured as a map rather than as a ranker
print(tables._cluster())8. One store or three
print(R.summary('mixed', ['genre','mode','encoder','strategy'], 'u_mrr', label='section MRR'))
print(R.summary('mixed', ['genre','mode','encoder','strategy'], 'cross_genre', label='share of top-10 hits from the wrong genre'))What it costs
print(R.cost_table())The tiers
See docs/rag_tiers.md for the tier definitions, the per-genre overrides, and the list of things that did not pay for themselves.