# configuration tiers


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

### What this measures

`litesearch` has accumulated choices: four chunk granularities, five
encoders, late chunking, a document tree, an entity graph, clustering, a
reranker, and two ways to preprocess an FTS query. Nothing in the
library says which combination to use, and the defaults are not the best
combination for any of the three genres tested here.

This notebook reads the results of `python -m evals.run` and turns them
into configuration tiers. The harness lives in
[`evals/`](https://github.com/Karthik777/litesearch/tree/main/evals)
rather than in this notebook, because it builds ~50 stores and takes
hours, but every table below is generated from its output, and the full
report with the tier definitions is in
[`docs/rag_tiers.md`](https://github.com/Karthik777/litesearch/blob/main/docs/rag_tiers.md).

**Three genres, chosen because they fail differently:**

<table>
<colgroup>
<col style="width: 33%" />
<col style="width: 33%" />
<col style="width: 33%" />
</colgroup>
<thead>
<tr>
<th>genre</th>
<th>what it is</th>
<th>how it breaks things</th>
</tr>
</thead>
<tbody>
<tr>
<td>regulatory</td>
<td>8 EU directives and regulations, 489 pages</td>
<td>real hierarchy, and enormous term repetition between documents</td>
</tr>
<tr>
<td>arxiv</td>
<td>12 papers, cs.* and astro-ph</td>
<td>headings do not survive PDF conversion; notation does not survive at
all</td>
</tr>
<tr>
<td>astrology</td>
<td>7 books printed 1822–1920, 1,275 pages</td>
<td>archaic prose, <code>CHAPTER XI</code> instead of markup, one book
with no headings</td>
</tr>
</tbody>
</table>

The astro-ph papers are in there on purpose: they talk about Mars,
conjunctions and ascending nodes without meaning any of it the way the
astrology books do, which is the cheapest available test of whether a
retriever separates genres or only matches words.

### Ground truth without an LLM

Every query comes from one sentence that occurs exactly once in the
corpus, so its passage and its section are known exactly. Five flavours
degrade the wording progressively, and all five are asked of the same
source sentences, so every comparison in this notebook is paired.

Scoring is keyed on **word 5-grams that are unique to one section**, at
three levels, passage, section, document. Nothing in the metric moves
when the chunk size does, which matters because chunk size is the
variable with the largest effect.

``` python
from evals import tables, report as R
from evals.queries import build as build_queries, FLAVOURS, overlap
print(tables._corpus())
```

``` python
qs = build_queries('astrology')
for fl in FLAVOURS: print(f'{fl:<11} {qs[3][fl]}')
```

### 1. Chunk granularity

The single largest effect in the whole evaluation, and the one place
where litesearch’s default is actively wrong:
[`chunk_markdown`](https://Karthik777.github.io/litesearch/data.html#chunk_markdown)
uses `FastChunker(chunk_size=4096)` and `add_doc` feeds it one node
segment at a time, so on pages of 1.7–3.5k characters the default
chunker almost never splits anything. `page` below is the default.

``` python
print(R.summary('grain', ['genre','chunking','encoder','strategy'], 'u_mrr', label='section MRR'))
```

### 2. Encoder

Five encoders spanning three orders of magnitude of cost per chunk, all
normalised to float16 and all measured on the same 4 CPU cores.

``` python
print(R.summary('encoder', ['genre','encoder','strategy'], 'u_mrr', label='section MRR'))
```

### 3. Retrieval strategy

`db.search` quotes each token of the query, which makes the FTS leg an
implicit AND over every token.
[`pre()`](https://Karthik777.github.io/litesearch/data.html#pre) exists
in the library to turn a query into keywords with wildcards and OR, but
`search` does not call it, and the caller has to know to.

``` python
print(R.summary('strategy', ['genre','encoder','strategy'], 'u_mrr', label='section MRR'))
```

### 4. Structure: flat, tree, and the headings PDF conversion throws away

``` python
print(R.summary('structure', ['genre','mode','encoder','strategy'], 'u_mrr', label='section MRR'))
```

### 5. Late chunking

``` python
print(R.summary('late', ['genre','mode','strategy'], 'u_mrr', label='section MRR'))
```

### 6. The graph leg

``` python
print(R.summary('graph', ['genre','strategy','graph_w'], 'u_mrr', label='section MRR'))
print(R.summary('graph_notopic', ['genre','mode','strategy'], 'u_mrr', label='same graph, topic nodes removed'))
```

### 7. Clustering, measured as a map rather than as a ranker

``` python
print(tables._cluster())
```

### 8. One store or three

``` python
print(R.summary('mixed', ['genre','mode','encoder','strategy'], 'u_mrr', label='section MRR'))
print(R.summary('mixed', ['genre','mode','encoder','strategy'], 'cross_genre', label='share of top-10 hits from the wrong genre'))
```

### What it costs

``` python
print(R.cost_table())
```

### The tiers

See
[`docs/rag_tiers.md`](https://github.com/Karthik777/litesearch/blob/main/docs/rag_tiers.md)
for the tier definitions, the per-genre overrides, and the list of
things that did not pay for themselves.