tree

Document structure — a navigable tree beside a litesearch store

What a tree buys

A chunk store answers “which 400 characters mention this”. A tree answers “which section”. It builds a node tree per document at ingest, links every chunk to a node, and adds three things to hybrid search: evidence that rolls up to sections, toc() and read() with no embeddings, and merged spans so a model sees contiguous text.

No language model anywhere. summarize= and build= take callables for one.

Detecting structure

Three signals in order: markdown headings, then chapter lines for scanned books, then fixed page windows as the floor. The floor keeps every document navigable.

struct_levels compacts heading levels rather than fixing them, so a document that only says “Article” gets articles at level 1. Unknown words rank by frequency: the rarer word names the bigger division.

detect_mode tries verse first. A Sanskrit source matches neither markdown nor chapter and fell through to window: GRETIL’s Manusmṛti collapsed 12 adhyāyas and 2,685 verses into one node.

detect_mode

def detect_mode(
    pages
)->str:

Which structural signal this document carries: markdown, verse, chapter or window.

struct_levels

def struct_levels(
    pages, max_levels:int=4
)->dict:

{heading word: level} for one document, as consecutive levels starting at 1.

summarize_extractive

def summarize_extractive(
    text:str, n:int=300
)->str:

First n clean characters. Cheap, deterministic, and the seam an LLM slots into via summarize=.

TreeNode

def TreeNode(
    seq:int, title:str, level:int, parent:int | None=None, page_start:int=0, page_end:int=0, segments:list=<factory>,
    children:list=<factory>, summary:str=''
)->None:

One section of a document. seq is its index in the flat list; parent is another seq.

build_tree returns a flat list. nodes[seq] is O(1), the parent link is an int, and it maps onto a SQL table with no serialiser. toc() builds the nesting on demand.

Verse mode reads headings too. GRETIL addresses verses by citation, but a stotra names sections (## dhyanam) and carries no citation. Reading only one gives the other a flat tree.

Citation depth is relative to the heading above it, because # depth starts wherever the markup starts and a citation’s hierarchy always starts at 1. Sharing one level space made TEI come out with empty adhyāya nodes.

_dedent_path drops a segment repeating the one before it, so a PDF whose H1 is its filename does not read Field Manual › Field Manual › Chapter 1 in every breadcrumb.

heading_path is embedded with the chunk and indexed for FTS. “The effects are severe” means nothing until you know which chapter said it.

heading_path

def heading_path(
    tree, # the node list from build_tree
    nd, # the TreeNode to describe
    title:str, # document title
    sep:str=' › ', max_len:int=200
)->str:

Doc › Part › Chapter for one node.

build_tree

def build_tree(
    pages, # [(page_no, text)] — markdown or plain text
    title:str='Document', # title of the root node
    summarize:NoneType=None, # callable(text)->str for node summaries (LLM goes here)
    window:int=8, # pages per node in `window` mode
    min_level:int=1, # floor for heading depth
    max_levels:int=4, # deepest node level kept
    mode:str=None, # force a structural mode instead of detecting one
)->list:

A flat list of TreeNode for a document (index = seq, root = 0).

tree = build_tree([(0, '# Saturn\n\nIntro text about the ringed planet.\n\n## Transits\n\nSade sati runs seven years.'),
                   (1, '## Remedies\n\nRecite on Saturdays.')], title='Jyotisha')
for n in tree: print(f'{n.seq}  lvl{n.level}  p{n.page_start}-{n.page_end}  {n.title!r:24} {n.summary[:40]!r}')
assert [n.title for n in tree] == ['Jyotisha', 'Saturn', 'Transits', 'Remedies']
assert tree[2].parent == 1 and tree[3].parent == 1, 'h2s hang off the h1, not off each other'
assert heading_path(tree, tree[2], 'Jyotisha') == 'Jyotisha › Saturn › Transits'

0  lvl0  p0-1  'Jyotisha'               'Saturn'
1  lvl1  p0-0  'Saturn'                 'Intro text about the ringed planet.'
2  lvl2  p0-0  'Transits'               'Sade sati runs seven years.'
3  lvl2  p1-1  'Remedies'               'Recite on Saturdays.'

# a skipped heading level must not nest siblings under each other. Both of these are ordinary
# documents: a package README whose h1 is lowercase (so the heading heuristic drops it) and leaves
# `##` as the shallowest heading, and a document that jumps from `#` straight to `###`.
_gap = build_tree([(0, '# litesearch\n\n> a tagline\n\n## Install\n\npip install\n\n'
                      '## Encoders\n\nencoders\n\n### Fallback\n\nhashing')], title='litesearch')
assert [n.title for n in _gap] == ['litesearch', 'Install', 'Encoders', 'Fallback']
assert [n.parent for n in _gap] == [None, 0, 0, 2], [n.parent for n in _gap]
assert heading_path(_gap, _gap[2], 'litesearch') == 'litesearch › Encoders'
assert heading_path(_gap, _gap[3], 'litesearch') == 'litesearch › Encoders › Fallback'

_jump = build_tree([(0, '# Doc\n\ntop\n\n### Deep One\n\na\n\n### Deep Two\n\nb')], title='Doc')
assert [n.parent for n in _jump] == [None, 0, 1, 1], [(n.title, n.level, n.parent) for n in _jump]

# a `#` comment inside a fenced block is a comment, not an h1 — otherwise a README's usage examples
# invent sections that adopt every real heading after them
_readme = '''# vishalakshi

> a tagline

## Harvest

```python
v.apis('https://shop/browse')
# [0] .../api/bff/products?page=1   records: 24
```

## Watches

watch what changes
'''
_fen = build_tree([(0, _readme)], title='vishalakshi')
assert [n.title for n in _fen] == ['vishalakshi', 'Harvest', 'Watches'], [n.title for n in _fen]
assert [n.parent for n in _fen] == [None, 0, 0]
assert 'records: 24' in _fen[1].text(), 'the fenced lines are still text, only never headings'
assert _md_stats([(0, _readme)]) == (3, 2)        # h1 + 2 h2; the comment is not a heading
assert detect_mode([(0, _readme)]) == 'markdown'

# an unclosed fence swallows the rest of the document rather than mis-reading it as structure
_open = build_tree([(0, '# Doc\n\n## Alpha\n\n```\n## Beta\n')], title='Doc')
assert [n.title for n in _open] == ['Doc', 'Doc', 'Alpha'], [n.title for n in _open]
assert '## Beta' in _open[2].text()

# a ````-fenced block quotes ``` blocks inside itself: only a fence at least as long closes it
_nest = '# Doc\n\n## Alpha\n\n````md\n```\n# Not a heading\n```\n````\n\n## Beta\n\nb'
assert [n.title for n in build_tree([(0, _nest)], title='Doc')] == ['Doc', 'Doc', 'Alpha', 'Beta']
# and the two fence characters do not close each other
assert list(_md_lines('~~~\n# x\n```\n# y\n~~~')) == [
    ('~~~', True), ('# x', True), ('```', True), ('# y', True), ('~~~', True)]

# a scanned book has chapter lines, not markdown; and a doc with neither still gets a tree
_book = build_tree([(i, f'CHAPTER {i+1}\n\nbody of chapter {i+1}') for i in range(4)], title='Saravali')
assert detect_mode([(i, f'CHAPTER {i+1}\n\nbody') for i in range(4)]) == 'chapter'
assert len(_book) == 5 and _book[1].title.startswith('CHAPTER 1')

_plain = build_tree([(i, f'page {i} of undifferentiated prose') for i in range(20)], title='Notes', window=8)
assert detect_mode([(i, 'prose') for i in range(20)]) == 'window'
assert len(_plain) == 4, [n.title for n in _plain]   # root + ceil(20/8) windows
assert _plain[1].title.startswith('Pages 1–8')

The tables

Two tables and three columns on the chunk store. The store stays the source of truth, so FTS5, the ANN index, db.search and the graph layer work on it unchanged.

docs    id · title · source · kind · pages · meta · added_at
nodes   id ('doc#seq') · doc_id · parent_id · level · seq · title ·
        page_start · page_end · summary · nchunks
store   content · embedding · metadata · doc_id · node_id · page · heading   [+ FTS5, +ANN]

Doc ids are content-addressed on (source, title), so re-adding a document is a no-op.

Database.get_tree

def get_tree(
    store:str='store', # chunk store the tree is built over
    prefix:str=None, # table prefix (default: '' for 'store', else '<store>_')
    ann:bool=True, # register an ANN index on the chunk store
    **kw
)->AttrDict: # extra typed columns for the chunk store

Create the docs/nodes tables and a node-aware chunk store. Idempotent; returns the tables. sections, read and breadcrumb all call this per query, so the DDL runs once per connection (Database.ensured) rather than once per call.

doc_id

def doc_id(
    source, title:str=''
)->str:

Content-addressed document id — re-adding the same source is a no-op, not a duplicate.

Ingestion

add_doc is the pipeline: tree, node-aware chunks, embed, store. It takes pages rather than a file, so acquisition stays elsewhere. It insists a chunk carries its heading path, embedded with the text and stored for FTS.

_pack_target is the size a chunker packs across segments to, 0 for one call per segment.

A chunker sees one segment at a time, and a segment carries a page number. GRETIL prints one verse per line and gretil_parse makes each verse a page, so ProseChunker’s 700-character target was applied to a 60-character verse and never reached: the Īśopaniṣad came out at 10 chunks through either chunker.

_node_chunks merges a chunk under min_chunk into its predecessor rather than dropping it.

Database.delete_doc

def delete_doc(
    did:str, store:str='store', prefix:str=None
):

Remove a document, its nodes and its chunks, dropping just those keys from the ANN index.

store_chunks

def store_chunks(
    store, # chunk store table
    chunks, # chunk dicts from `_node_chunks`, from one document or many
    emb_fn:NoneType=None, # embedder: list[str] -> vectors
    with_heading:bool=True, # embed each chunk together with its heading path
):

Embed a batch of chunks and upsert them. The batch may span documents, which is the point.

Database.add_doc

def add_doc(
    pages, # [(page_no, text)] — or a single string
    title:str, # document title
    source:str=None, # path or url (defaults to the title)
    kind:str='text', # 'pdf' | 'web' | 'md' | 'code' | anything you filter on
    store:str='store', # chunk store
    prefix:str=None, # tree table prefix
    emb_fn:NoneType=None, # embedder: list[str] -> vectors
    chunker:NoneType=None, # chonkie chunker (default: FastChunker via chunk_markdown)
    summarize:NoneType=None, # callable(text)->str for node summaries
    with_heading:bool=True, # embed each chunk together with its heading path
    meta:dict=None, # arbitrary json metadata for the doc row
    force:bool=False, # re-ingest a document already present
    index:bool=True, # mirror this document's chunks into the ANN index
    mode:str=None, # force a build_tree structural mode instead of detecting one
    meta_fn:NoneType=None, # (chunk text) -> dict of facets stored as the chunk's `metadata`
    defer:list=None, # collect chunks here instead of embedding and storing them
)->dict:

Ingest one document: build its tree, chunk it per node, embed and store.

add_file is the one-liner for the common case. It only knows about documents, PDFs, markdown, plain text, notebooks. Source code is a different tree (module › class › function) and belongs to kosha, which builds it from the AST rather than from headings.

`add_file`

Ingest one document file. PDFs page through pdf_parse; everything else is one page of text.

A registered Profile wins over the extension table: it is the only thing that can tell a TEI edition from any other .xml, and it carries the chunker, tree mode and per-chunk facets that format needs.

profile= names one directly, which is the only way to reach a profile that cannot be detected: sanskrit_prose shares every signal with sanskrit_verse and differs only in its chunk budget, so nothing but the caller can tell them apart.

Database.add_file

def add_file(
    path, # file to ingest
    title:str=None, # defaults to a prettified filename
    store:str='store', prefix:str=None, kind:str=None, # overrides the kind inferred from the extension
    out_path:NoneType=None, # dir for extracted PDF images; defaults to `assets/<stem>` beside the db
    profile:str=None, # force a registered Profile by name instead of detecting one
    **kw
)->dict: # forwarded to add_doc (emb_fn, chunker, summarize, force, ...)

Ingest one document file. PDFs page through pdf_parse; everything else is one page of text.

Database.assets

def assets(
    name:str=None
)->Path:

Where extracted assets (PDF images) go: beside the database file, never the working directory.

add_dir keeps FTS triggers active by default. bulk=True removes them and rebuilds FTS after the load. Keyword search stays stale until that rebuild.

Parsing can use processes. Writes stay in the parent process because the connection, FTS index and HNSW index are shared state. Eight PDFs spent 3.55 seconds of a 5.31-second walk in parsing. A process pool helps there. Reading 150 Markdown files took 2.7 milliseconds, so n_workers=None does not start a pool for cheap formats.

files= ingests an explicit list. This supports routing files from one directory to different stores. Each call writes one store because its embedder and ANN index belong to that store.

The parent resolves each file’s Profile before starting workers. A worker without that registered profile returns the file for parsing in the parent rather than reading it as plain text.

Database.add_dir

def add_dir(
    dir:NoneType=None, # directory to walk; ignored when `files` is given
    types:str='.pdf,.md,.markdown,.txt,.rst,.ipynb,.xml,.tei,.htm,.html,.conllu', # comma-separated extensions to ingest
    store:str='store', prefix:str=None,
    n_workers:int=None, # parse workers; 0 is serial, None picks by parse-heavy file count
    embed_batch:int=2000, # chunks embedded and written per flush; 0 writes per document
    files:NoneType=None, # ingest exactly these paths instead of walking `dir`
    bulk:bool=False, # rebuild FTS once after an offline load; FTS is stale during the load
    **kw
)->list: # forwarded to add_doc

Ingest every document under a directory tree. Already-ingested sources are skipped, not duplicated.

from tempfile import TemporaryDirectory
from litesearch.core import database
with TemporaryDirectory() as d:
    Path(d, 'a.md').write_text('# Alpha\n\npolonium report')
    x = database(Path(d)/'test.db', sem_search=False)
    x.add_dir(d, types='md')
    assert x.t.store.fts_search('polonium')

with TemporaryDirectory() as d:
    Path(d, 'a.md').write_text('# Alpha\n\npolonium report')
    Path(d, 'b.md').write_text('# Beta\n\nthorium report')

    x = database(Path(d)/'test.db', sem_search=False)
    x.add_dir(d, types='md', bulk=True)

    assert len(x.t.store.fts_search('polonium')) == 1
    assert len(x.t.store.fts_search('thorium')) == 1

    # Triggers were restored after the rebuild.
    x.t.store.insert({'content': 'later radium report'})
    assert len(x.t.store.fts_search('radium')) == 1

Reading the structure

toc and read compute no embeddings. An agent that already wants “the chapter on remedies” reads the table of contents and opens the node.

Database.read

def read(
    node_id:str, # 'doc#seq', as returned by toc() or sections()
    store:str='store', prefix:str=None, max_chars:int=20000, # cap on the assembled text
    children:bool=True, # append the text of child nodes too
)->dict:

One whole section — the unit an agent should read instead of guessing from fragments.

Database.toc

def toc(
    doc:str=None, # doc id, or a substring of the title (None = every document)
    store:str='store', prefix:str=None, max_depth:int=3, # deepest level included
    summaries:bool=True, # include node summaries
)->list:

The document tree as nested dicts — titles, page ranges, summaries. No embeddings touched.

Retrieval that knows about structure

Three refinements over db.search.

Adaptive fusion reweights the two legs when a query is quoted or identifier-heavy, or when FTS returns nothing. Off by default: neither trigger fired across 300 natural-language queries.

Spans merge adjacent hits inside a node, so a model gets one passage instead of two halves.

Section rollup scores a node by its best hit. Summing every hit’s RRF mass is a length prior: on 150 known-item queries over 486 pages of legislation it cost 0.07 MRR on verbatim queries and 0.16 on degraded ones. score='sum' stays available for uniform section lengths.

adaptive_weights is three rules, not a model: a quoted phrase and a rare identifier are literal requests, an empty leg cannot vote, everything else ties. Off by default, and a replacement rule leaning on FTS coverage measured 0.425 against 0.496 MRR on degraded queries.

merge_spans keeps the best rank and score of its members, so ordering is unchanged.

sections uses score='max'. mean is within noise of it.

Database.sections

def sections(
    q:str, # query string
    emb:bytes, # query embedding
    limit:int=5, # sections to return
    per:int=3, # snippets kept per section
    store:str='store', prefix:str=None, fanout:int=8, # chunk hits gathered before rolling up
    score:str='max', # how a section scores from its hits: max | mean | sum
    rerank:bool=False, # reorder the chunk hits before they are grouped into sections
    **kw
)->list: # forwarded to doc_search

Ranked sections, not chunks: hits grouped by node, each with a read handle.

Database.doc_search

def doc_search(
    q:str, # query string
    emb:bytes, # query embedding
    columns:list=None, # extra columns (node_id, page, heading, doc_id always included)
    limit:int=10, store:str='store', prefix:str=None, spans:bool=True, # merge adjacent hits inside a node
    gap:int=1, # span merge tolerance, in pages
    adaptive:bool=False, # weight the RRF legs by query shape (measured inert — see below)
    rrf_k:int=60, rerank:bool=False, # reorder the candidates with a flashrank cross-encoder
    **kw
)->list: # forwarded to Database.search

Hybrid search over a node-aware store: span merging and a breadcrumb per hit.

merge_spans

def merge_spans(
    hits, # ranked hits carrying node_id + page
    gap:int=1, # merge hits at most this many pages apart
)->list:

Collapse hits that are adjacent inside the same node into one span.

adaptive_weights

def adaptive_weights(
    q:str, # the raw query
    fts:list, # the FTS leg's results
    vec:list, # the vector leg's results
)->tuple:

(fts_weight, vec_weight) for RRF, from cheap signals in the query and the legs.

context composes one retrieval: the operative sections plus what they connect to. ann, rerank and fts_pre reach Database.search through it.

Database.context

def context(
    q:str, # query string
    emb:bytes, # query embedding
    store:str='store', prefix:str=None, sections:int=6, # operative sections returned
    per:int=3, # snippets kept per section
    graph:bool=False, # opt in to graph-reached related sections (see the note below)
    vector:bool=True, # include embedding-nearest related sections
    related:int=8, # max related sections
    max_read:int=6000, # chars of assembled text per operative section
    tree_ctx:bool=True, # attach each section's parent/siblings/children
    graph_w:float=0.6, **kw
):

One composed retrieval over a document tree: the operative sections plus what they connect to.

Database.node_context

def node_context(
    node_id:str, # a 'doc#seq' node id
    store:str='store', prefix:str=None
):

Where a section sits: its parent, siblings and children — a hit as a provision in a structure.

End to end

A three-chapter document, a hash embedder so the notebook runs offline, and every call in the module.

import numpy as np, hashlib
from litesearch.core import database
from litesearch.utils import hash_embed

DOC = """# Saturn

Saturn is the slowest of the classical grahas.

## Sade Sati

Sade sati is the seven and a half year period when Saturn transits the twelfth, first and second
houses from the natal moon. It is counted in three phases of roughly thirty months each.

The middle phase, with Saturn over the moon itself, is held to be the heaviest.

## Remedies

Recitation on Saturdays is the common prescription. Donation of black sesame is another.

# Jupiter

Jupiter is the great benefic and moves through one sign each year.

## Transits

Jupiter's return to its natal sign happens roughly every twelve years."""

db  = database()
enc = lambda ts, **kw: hash_embed(ts, 256)
db.get_tree('store')
print(db.add_doc(DOC, title='Grahas', source='notes/grahas.md', kind='md', emb_fn=enc))

{'doc_id': '5a991f283775ae0d', 'title': 'Grahas', 'kind': 'md', 'nodes': 6, 'chunks': 5}

# the tree came out of the headings, and every chunk knows which node it lives in
t = db.toc('Grahas')[0]
def show(n, d=0):
    print('  '*d + f"{n['title']:<14} chunks={n['nchunks']}  {n['id']}")
    for c in n.get('children', []): show(c, d+1)
show(t['tree'])
assert [c['title'] for c in t['tree']['children']] == ['Saturn', 'Jupiter']
assert db.breadcrumb(t['tree']['children'][0]['children'][0]['id']) == 'Grahas › Saturn › Sade Sati'

Grahas         chunks=0  5a991f283775ae0d#0
  Saturn         chunks=1  5a991f283775ae0d#1
    Sade Sati      chunks=1  5a991f283775ae0d#2
    Remedies       chunks=1  5a991f283775ae0d#3
  Jupiter        chunks=1  5a991f283775ae0d#4
    Transits       chunks=1  5a991f283775ae0d#5

# read() returns the section, not a fragment -- and includes its children
sec = first(t['tree']['children'][0]['children'], lambda c: c['title'] == 'Sade Sati')
r = db.read(sec['id'])
print(r['breadcrumb'], '|', r['pages'], '|', len(r['text']), 'chars')
assert 'thirty months' in r['text'] and r['breadcrumb'].endswith('Sade Sati')

Grahas › Saturn › Sade Sati | (0, 0) | 266 chars

# hybrid search, but every hit is placed in the document
q  = 'how long does the saturn transit last'
hits = db.doc_search(q, enc([q])[0].tobytes(), limit=3, dtype=np.float16)
for h in hits: print(f"{h['_rrf_score']:.4f}  {h['breadcrumb']:<28} {h['content'][:60]!r}")
assert hits and all(h['breadcrumb'] for h in hits)

# sections() rolls the same evidence up a level and hands back the next call
secs = db.sections(q, enc([q])[0].tobytes(), limit=2, dtype=np.float16)
for s in secs: print(f"{s['score']:.4f}  {s['breadcrumb']:<28} {s['read']}")
assert secs and secs[0]['read'].startswith('read(')

0.0333  Grahas › Saturn › Sade Sati  'Sade sati is the seven and a half year period when Saturn tr'
0.0328  Grahas › Saturn              'Saturn is the slowest of the classical grahas.'
0.0161  Grahas › Saturn › Remedies   'Recitation on Saturdays is the common prescription. Donation'
0.0333  Grahas › Saturn › Sade Sati  read('5a991f283775ae0d#2')
0.0328  Grahas › Saturn              read('5a991f283775ae0d#1')

_rq  = 'how long does the saturn transit last'
_rqv = enc([_rq])[0].tobytes()
_plain = db.doc_search(_rq, _rqv, limit=3, dtype=np.float16)
_rr    = db.doc_search(_rq, _rqv, limit=3, dtype=np.float16, rerank=True)
assert len(_rr) <= 3 and all(h['breadcrumb'] for h in _rr)
assert [h['_rrf_score'] for h in _rr] == [1.0/(60+i) for i in range(len(_rr))]
assert [h['_rrf_score'] for h in _plain] != [1.0/(60+i) for i in range(len(_plain))]
# and the roll-up inherits it: sections forwards rerank to the chunk hits it groups
_srr = db.sections(_rq, _rqv, limit=2, dtype=np.float16, rerank=True)
assert _srr and all(s['breadcrumb'] for s in _srr)
print('reranked:', [h['content'][:40] for h in _rr])

INFO:flashrank.Ranker:Downloading ms-marco-TinyBERT-L-2-v2...

ms-marco-TinyBERT-L-2-v2.zip:   0%|          | 0.00/3.26M [00:00<?, ?iB/s]
ms-marco-TinyBERT-L-2-v2.zip:   1%|          | 24.0k/3.26M [00:00<00:17, 189kiB/s]
ms-marco-TinyBERT-L-2-v2.zip:   2%|▏         | 64.0k/3.26M [00:00<00:11, 304kiB/s]
ms-marco-TinyBERT-L-2-v2.zip:   4%|▍         | 136k/3.26M [00:00<00:06, 482kiB/s] 
ms-marco-TinyBERT-L-2-v2.zip:   8%|▊         | 272k/3.26M [00:00<00:03, 816kiB/s]
ms-marco-TinyBERT-L-2-v2.zip:  15%|█▌        | 512k/3.26M [00:00<00:02, 1.37MiB/s]
ms-marco-TinyBERT-L-2-v2.zip:  30%|██▉       | 992k/3.26M [00:00<00:00, 2.52MiB/s]
ms-marco-TinyBERT-L-2-v2.zip:  55%|█████▍    | 1.78M/3.26M [00:00<00:00, 4.42MiB/s]
ms-marco-TinyBERT-L-2-v2.zip: 100%|██████████| 3.26M/3.26M [00:00<00:00, 4.08MiB/s]

reranked: ['Sade sati is the seven and a half year p', 'Saturn is the slowest of the classical g', "Jupiter's return to its natal sign happe"]

# adaptive fusion: a quoted phrase leans on FTS, an empty FTS leg hands the vote to vectors
assert adaptive_weights('"sade sati"', [1], [1])[0] > adaptive_weights('how long does it last', [1], [1])[0]
assert adaptive_weights('anything', [], [1]) == (0.0, 1.0)
assert adaptive_weights('anything', [1], []) == (1.0, 0.0)

# spans: two hits one page apart in the same node become one passage
_h = [dict(content='first half', node_id='d#1', page=3, _rrf_score=0.02, rowid=1),
      dict(content='second half', node_id='d#1', page=4, _rrf_score=0.01, rowid=2),
      dict(content='elsewhere',   node_id='d#9', page=40, _rrf_score=0.005, rowid=3)]
_m = merge_spans(_h)
assert len(_m) == 2 and _m[0]['_nspan'] == 2 and 'second half' in _m[0]['content'], _m
assert _m[0]['_rrf_score'] == 0.02, 'a merged span keeps its best score, it does not accumulate'

# ingestion is content-addressed, and delete_doc leaves nothing behind
again = db.add_doc(DOC, title='Grahas', source='notes/grahas.md', emb_fn=enc)
assert 'skipped' in again, again
did = t['doc_id']
n_before = len(db.t.store())
db.delete_doc(did)
assert len(db.t.store()) == 0 and db.toc() == [] and n_before > 0
print(f'{n_before} chunks removed with the document')

5 chunks removed with the document

Not done yet

No LLM. summarize= takes a callable. A summary per node is what toc() shows an agent deciding where to read.
Chunking is chunk_markdown. A cost-model splitter produces better boundaries on prose. The seam is chunker=.
Chunking is where the gains are. In 07_doc_eval, node-scoped chunks lifted MRR from 0.330 to 0.522 on degraded queries, more than everything else here combined.

import tempfile
_d = Path(tempfile.mkdtemp())
(_d/'ratios.md').write_text('# Ratios\n\n## Liquidity\n\nCurrent ratio is current assets over current liabilities.')
(_d/'notes.txt').write_text('A plain text file with no headings at all, ingested as one windowed node.')
_fdb = database()
_fdb.get_tree('store')
print(_fdb.add_dir(_d, emb_fn=enc))
assert {d['title'] for d in _fdb.toc()} == {'ratios', 'notes'}
_rt = _fdb.toc('ratios')[0]['tree']
assert [n['title'] for n in _rt['children']] == ['Ratios']            # the `# Ratios` heading
assert [n['title'] for n in _rt['children'][0]['children']] == ['Liquidity']
assert _fdb.add_file(_d/'ratios.md', emb_fn=enc).get('skipped'), 're-ingest is a no-op'
assert 'current liabilities' in _fdb.read(_fdb.toc('ratios')[0]['tree']['id'])['text']

[{'doc_id': '1e8d58569295fb93', 'title': 'notes', 'kind': 'txt', 'nodes': 2, 'chunks': 1}, {'doc_id': 'b4432a03f1d158da', 'title': 'ratios', 'kind': 'md', 'nodes': 3, 'chunks': 1}]