# tree ## What a tree buys A chunk store answers “which 400 characters mention this”. A tree answers “which section”. It builds a node tree per document at ingest, links every chunk to a node, and adds three things to hybrid search: evidence that rolls up to sections, `toc()` and `read()` with no embeddings, and merged spans so a model sees contiguous text. No language model anywhere. `summarize=` and `build=` take callables for one. ## Detecting structure Three signals in order: markdown headings, then chapter lines for scanned books, then fixed page windows as the floor. The floor keeps every document navigable. [`struct_levels`](https://Karthik777.github.io/litesearch/tree.html#struct_levels) compacts heading levels rather than fixing them, so a document that only says “Article” gets articles at level 1. Unknown words rank by frequency: the rarer word names the bigger division. [`detect_mode`](https://Karthik777.github.io/litesearch/tree.html#detect_mode) tries `verse` first. A Sanskrit source matches neither markdown nor chapter and fell through to `window`: GRETIL’s Manusmṛti collapsed 12 adhyāyas and 2,685 verses into one node. ------------------------------------------------------------------------ source ### detect_mode ``` python def detect_mode( pages )->str: ``` *Which structural signal this document carries: `markdown`, `verse`, `chapter` or `window`.* ------------------------------------------------------------------------ source ### struct_levels ``` python def struct_levels( pages, max_levels:int=4 )->dict: ``` *`{heading word: level}` for one document, as consecutive levels starting at 1.* ------------------------------------------------------------------------ source ### summarize_extractive ``` python def summarize_extractive( text:str, n:int=300 )->str: ``` *First `n` clean characters. Cheap, deterministic, and the seam an LLM slots into via `summarize=`.* ------------------------------------------------------------------------ source ### TreeNode ``` python def TreeNode( seq:int, title:str, level:int, parent:int | None=None, page_start:int=0, page_end:int=0, segments:list=, children:list=, summary:str='' )->None: ``` *One section of a document. `seq` is its index in the flat list; `parent` is another `seq`.* [`build_tree`](https://Karthik777.github.io/litesearch/tree.html#build_tree) returns a flat list. `nodes[seq]` is O(1), the parent link is an int, and it maps onto a SQL table with no serialiser. `toc()` builds the nesting on demand. Verse mode reads headings too. GRETIL addresses verses by citation, but a stotra names sections (`## dhyanam`) and carries no citation. Reading only one gives the other a flat tree. Citation depth is relative to the heading above it, because `#` depth starts wherever the markup starts and a citation’s hierarchy always starts at 1. Sharing one level space made TEI come out with empty adhyāya nodes. [`_dedent_path`](https://Karthik777.github.io/litesearch/tree.html#_dedent_path) drops a segment repeating the one before it, so a PDF whose H1 is its filename does not read `Field Manual › Field Manual › Chapter 1` in every breadcrumb. [`heading_path`](https://Karthik777.github.io/litesearch/tree.html#heading_path) is embedded with the chunk and indexed for FTS. “The effects are severe” means nothing until you know which chapter said it. ------------------------------------------------------------------------ source ### heading_path ``` python def heading_path( tree, # the node list from build_tree nd, # the TreeNode to describe title:str, # document title sep:str=' › ', max_len:int=200 )->str: ``` *`Doc › Part › Chapter` for one node.* ------------------------------------------------------------------------ source ### build_tree ``` python def build_tree( pages, # [(page_no, text)] — markdown or plain text title:str='Document', # title of the root node summarize:NoneType=None, # callable(text)->str for node summaries (LLM goes here) window:int=8, # pages per node in `window` mode min_level:int=1, # floor for heading depth max_levels:int=4, # deepest node level kept mode:str=None, # force a structural mode instead of detecting one )->list: ``` *A flat list of [`TreeNode`](https://Karthik777.github.io/litesearch/tree.html#treenode) for a document (index = seq, root = 0).* ``` python tree = build_tree([(0, '# Saturn\n\nIntro text about the ringed planet.\n\n## Transits\n\nSade sati runs seven years.'), (1, '## Remedies\n\nRecite on Saturdays.')], title='Jyotisha') for n in tree: print(f'{n.seq} lvl{n.level} p{n.page_start}-{n.page_end} {n.title!r:24} {n.summary[:40]!r}') assert [n.title for n in tree] == ['Jyotisha', 'Saturn', 'Transits', 'Remedies'] assert tree[2].parent == 1 and tree[3].parent == 1, 'h2s hang off the h1, not off each other' assert heading_path(tree, tree[2], 'Jyotisha') == 'Jyotisha › Saturn › Transits' ``` 0 lvl0 p0-1 'Jyotisha' 'Saturn' 1 lvl1 p0-0 'Saturn' 'Intro text about the ringed planet.' 2 lvl2 p0-0 'Transits' 'Sade sati runs seven years.' 3 lvl2 p1-1 'Remedies' 'Recite on Saturdays.' ``` python # a skipped heading level must not nest siblings under each other. Both of these are ordinary # documents: a package README whose h1 is lowercase (so the heading heuristic drops it) and leaves # `##` as the shallowest heading, and a document that jumps from `#` straight to `###`. _gap = build_tree([(0, '# litesearch\n\n> a tagline\n\n## Install\n\npip install\n\n' '## Encoders\n\nencoders\n\n### Fallback\n\nhashing')], title='litesearch') assert [n.title for n in _gap] == ['litesearch', 'Install', 'Encoders', 'Fallback'] assert [n.parent for n in _gap] == [None, 0, 0, 2], [n.parent for n in _gap] assert heading_path(_gap, _gap[2], 'litesearch') == 'litesearch › Encoders' assert heading_path(_gap, _gap[3], 'litesearch') == 'litesearch › Encoders › Fallback' _jump = build_tree([(0, '# Doc\n\ntop\n\n### Deep One\n\na\n\n### Deep Two\n\nb')], title='Doc') assert [n.parent for n in _jump] == [None, 0, 1, 1], [(n.title, n.level, n.parent) for n in _jump] ``` ```` python # a `#` comment inside a fenced block is a comment, not an h1 — otherwise a README's usage examples # invent sections that adopt every real heading after them _readme = '''# vishalakshi > a tagline ## Harvest ```python v.apis('https://shop/browse') # [0] .../api/bff/products?page=1 records: 24 ``` ## Watches watch what changes ''' _fen = build_tree([(0, _readme)], title='vishalakshi') assert [n.title for n in _fen] == ['vishalakshi', 'Harvest', 'Watches'], [n.title for n in _fen] assert [n.parent for n in _fen] == [None, 0, 0] assert 'records: 24' in _fen[1].text(), 'the fenced lines are still text, only never headings' assert _md_stats([(0, _readme)]) == (3, 2) # h1 + 2 h2; the comment is not a heading assert detect_mode([(0, _readme)]) == 'markdown' # an unclosed fence swallows the rest of the document rather than mis-reading it as structure _open = build_tree([(0, '# Doc\n\n## Alpha\n\n```\n## Beta\n')], title='Doc') assert [n.title for n in _open] == ['Doc', 'Doc', 'Alpha'], [n.title for n in _open] assert '## Beta' in _open[2].text() # a ````-fenced block quotes ``` blocks inside itself: only a fence at least as long closes it _nest = '# Doc\n\n## Alpha\n\n````md\n```\n# Not a heading\n```\n````\n\n## Beta\n\nb' assert [n.title for n in build_tree([(0, _nest)], title='Doc')] == ['Doc', 'Doc', 'Alpha', 'Beta'] # and the two fence characters do not close each other assert list(_md_lines('~~~\n# x\n```\n# y\n~~~')) == [ ('~~~', True), ('# x', True), ('```', True), ('# y', True), ('~~~', True)] ```` ``` python # a scanned book has chapter lines, not markdown; and a doc with neither still gets a tree _book = build_tree([(i, f'CHAPTER {i+1}\n\nbody of chapter {i+1}') for i in range(4)], title='Saravali') assert detect_mode([(i, f'CHAPTER {i+1}\n\nbody') for i in range(4)]) == 'chapter' assert len(_book) == 5 and _book[1].title.startswith('CHAPTER 1') _plain = build_tree([(i, f'page {i} of undifferentiated prose') for i in range(20)], title='Notes', window=8) assert detect_mode([(i, 'prose') for i in range(20)]) == 'window' assert len(_plain) == 4, [n.title for n in _plain] # root + ceil(20/8) windows assert _plain[1].title.startswith('Pages 1–8') ``` ## The tables Two tables and three columns on the chunk store. The store stays the source of truth, so FTS5, the ANN index, `db.search` and the graph layer work on it unchanged. docs id · title · source · kind · pages · meta · added_at nodes id ('doc#seq') · doc_id · parent_id · level · seq · title · page_start · page_end · summary · nchunks store content · embedding · metadata · doc_id · node_id · page · heading [+ FTS5, +ANN] Doc ids are content-addressed on `(source, title)`, so re-adding a document is a no-op. ------------------------------------------------------------------------ source ### Database.get_tree ``` python def get_tree( store:str='store', # chunk store the tree is built over prefix:str=None, # table prefix (default: '' for 'store', else '_') ann:bool=True, # register an ANN index on the chunk store **kw )->AttrDict: # extra typed columns for the chunk store ``` *Create the docs/nodes tables and a node-aware chunk store. Idempotent; returns the tables.* `sections`, `read` and `breadcrumb` all call this per query, so the DDL runs once per connection ([`Database.ensured`](https://Karthik777.github.io/litesearch/core.html#database.ensured)) rather than once per call. ------------------------------------------------------------------------ source ### doc_id ``` python def doc_id( source, title:str='' )->str: ``` *Content-addressed document id — re-adding the same source is a no-op, not a duplicate.* ## Ingestion `add_doc` is the pipeline: tree, node-aware chunks, embed, store. It takes `pages` rather than a file, so acquisition stays elsewhere. It insists a chunk carries its `heading` path, embedded with the text and stored for FTS. [`_pack_target`](https://Karthik777.github.io/litesearch/tree.html#_pack_target) is the size a chunker packs across segments to, 0 for one call per segment. A chunker sees one segment at a time, and a segment carries a page number. GRETIL prints one verse per line and `gretil_parse` makes each verse a page, so `ProseChunker`’s 700-character target was applied to a 60-character verse and never reached: the Īśopaniṣad came out at 10 chunks through either chunker. [`_node_chunks`](https://Karthik777.github.io/litesearch/tree.html#_node_chunks) merges a chunk under `min_chunk` into its predecessor rather than dropping it. ------------------------------------------------------------------------ source ### Database.delete_doc ``` python def delete_doc( did:str, store:str='store', prefix:str=None ): ``` *Remove a document, its nodes and its chunks, dropping just those keys from the ANN index.* ------------------------------------------------------------------------ source ### store_chunks ``` python def store_chunks( store, # chunk store table chunks, # chunk dicts from `_node_chunks`, from one document or many emb_fn:NoneType=None, # embedder: list[str] -> vectors with_heading:bool=True, # embed each chunk together with its heading path ): ``` *Embed a batch of chunks and upsert them. The batch may span documents, which is the point.* ------------------------------------------------------------------------ source ### Database.add_doc ``` python def add_doc( pages, # [(page_no, text)] — or a single string title:str, # document title source:str=None, # path or url (defaults to the title) kind:str='text', # 'pdf' | 'web' | 'md' | 'code' | anything you filter on store:str='store', # chunk store prefix:str=None, # tree table prefix emb_fn:NoneType=None, # embedder: list[str] -> vectors chunker:NoneType=None, # chonkie chunker (default: FastChunker via chunk_markdown) summarize:NoneType=None, # callable(text)->str for node summaries with_heading:bool=True, # embed each chunk together with its heading path meta:dict=None, # arbitrary json metadata for the doc row force:bool=False, # re-ingest a document already present index:bool=True, # mirror this document's chunks into the ANN index mode:str=None, # force a build_tree structural mode instead of detecting one meta_fn:NoneType=None, # (chunk text) -> dict of facets stored as the chunk's `metadata` defer:list=None, # collect chunks here instead of embedding and storing them )->dict: ``` *Ingest one document: build its tree, chunk it per node, embed and store.* `add_file` is the one-liner for the common case. It only knows about documents, PDFs, markdown, plain text, notebooks. Source code is a different tree (module › class › function) and belongs to `kosha`, which builds it from the AST rather than from headings. #### `add_file` Ingest one document file. PDFs page through [`pdf_parse`](https://Karthik777.github.io/litesearch/data.html#pdf_parse); everything else is one page of text. A registered [`Profile`](https://Karthik777.github.io/litesearch/data.html#profile) wins over the extension table: it is the only thing that can tell a TEI edition from any other `.xml`, and it carries the chunker, tree mode and per-chunk facets that format needs. `profile=` names one directly, which is the only way to reach a profile that cannot be detected: `sanskrit_prose` shares every signal with `sanskrit_verse` and differs only in its chunk budget, so nothing but the caller can tell them apart. ------------------------------------------------------------------------ source ### Database.add_file ``` python def add_file( path, # file to ingest title:str=None, # defaults to a prettified filename store:str='store', prefix:str=None, kind:str=None, # overrides the kind inferred from the extension out_path:NoneType=None, # dir for extracted PDF images; defaults to `assets/` beside the db profile:str=None, # force a registered Profile by name instead of detecting one **kw )->dict: # forwarded to add_doc (emb_fn, chunker, summarize, force, ...) ``` *Ingest one document file. PDFs page through [`pdf_parse`](https://Karthik777.github.io/litesearch/data.html#pdf_parse); everything else is one page of text.* ------------------------------------------------------------------------ source ### Database.assets ``` python def assets( name:str=None )->Path: ``` *Where extracted assets (PDF images) go: beside the database file, never the working directory.* `add_dir` keeps FTS triggers active by default. `bulk=True` removes them and rebuilds FTS after the load. Keyword search stays stale until that rebuild. Parsing can use processes. Writes stay in the parent process because the connection, FTS index and HNSW index are shared state. Eight PDFs spent 3.55 seconds of a 5.31-second walk in parsing. A process pool helps there. Reading 150 Markdown files took 2.7 milliseconds, so `n_workers=None` does not start a pool for cheap formats. `files=` ingests an explicit list. This supports routing files from one directory to different stores. Each call writes one store because its embedder and ANN index belong to that store. The parent resolves each file’s [`Profile`](https://Karthik777.github.io/litesearch/data.html#profile) before starting workers. A worker without that registered profile returns the file for parsing in the parent rather than reading it as plain text. ------------------------------------------------------------------------ source ### Database.add_dir ``` python def add_dir( dir:NoneType=None, # directory to walk; ignored when `files` is given types:str='.pdf,.md,.markdown,.txt,.rst,.ipynb,.xml,.tei,.htm,.html,.conllu', # comma-separated extensions to ingest store:str='store', prefix:str=None, n_workers:int=None, # parse workers; 0 is serial, None picks by parse-heavy file count embed_batch:int=2000, # chunks embedded and written per flush; 0 writes per document files:NoneType=None, # ingest exactly these paths instead of walking `dir` bulk:bool=False, # rebuild FTS once after an offline load; FTS is stale during the load **kw )->list: # forwarded to add_doc ``` *Ingest every document under a directory tree. Already-ingested sources are skipped, not duplicated.* ``` python from tempfile import TemporaryDirectory from litesearch.core import database with TemporaryDirectory() as d: Path(d, 'a.md').write_text('# Alpha\n\npolonium report') x = database(Path(d)/'test.db', sem_search=False) x.add_dir(d, types='md') assert x.t.store.fts_search('polonium') ``` ``` python with TemporaryDirectory() as d: Path(d, 'a.md').write_text('# Alpha\n\npolonium report') Path(d, 'b.md').write_text('# Beta\n\nthorium report') x = database(Path(d)/'test.db', sem_search=False) x.add_dir(d, types='md', bulk=True) assert len(x.t.store.fts_search('polonium')) == 1 assert len(x.t.store.fts_search('thorium')) == 1 # Triggers were restored after the rebuild. x.t.store.insert({'content': 'later radium report'}) assert len(x.t.store.fts_search('radium')) == 1 ``` ## Reading the structure `toc` and `read` compute no embeddings. An agent that already wants “the chapter on remedies” reads the table of contents and opens the node. ------------------------------------------------------------------------ source ### Database.read ``` python def read( node_id:str, # 'doc#seq', as returned by toc() or sections() store:str='store', prefix:str=None, max_chars:int=20000, # cap on the assembled text children:bool=True, # append the text of child nodes too )->dict: ``` *One whole section — the unit an agent should read instead of guessing from fragments.* ------------------------------------------------------------------------ source ### Database.breadcrumb ``` python def breadcrumb( node_id:str, store:str='store', prefix:str=None, sep:str=' › ' )->str: ``` *The path from the document down to a node: `Book › Chapter › Section`.* ------------------------------------------------------------------------ source ### Database.toc ``` python def toc( doc:str=None, # doc id, or a substring of the title (None = every document) store:str='store', prefix:str=None, max_depth:int=3, # deepest level included summaries:bool=True, # include node summaries )->list: ``` *The document tree as nested dicts — titles, page ranges, summaries. No embeddings touched.* ## Retrieval that knows about structure Three refinements over `db.search`. **Adaptive fusion** reweights the two legs when a query is quoted or identifier-heavy, or when FTS returns nothing. Off by default: neither trigger fired across 300 natural-language queries. **Spans** merge adjacent hits inside a node, so a model gets one passage instead of two halves. **Section rollup** scores a node by its best hit. Summing every hit’s RRF mass is a length prior: on 150 known-item queries over 486 pages of legislation it cost 0.07 MRR on verbatim queries and 0.16 on degraded ones. `score='sum'` stays available for uniform section lengths. [`adaptive_weights`](https://Karthik777.github.io/litesearch/tree.html#adaptive_weights) is three rules, not a model: a quoted phrase and a rare identifier are literal requests, an empty leg cannot vote, everything else ties. Off by default, and a replacement rule leaning on FTS coverage measured 0.425 against 0.496 MRR on degraded queries. [`merge_spans`](https://Karthik777.github.io/litesearch/tree.html#merge_spans) keeps the best rank and score of its members, so ordering is unchanged. `sections` uses `score='max'`. `mean` is within noise of it. ------------------------------------------------------------------------ source ### Database.sections ``` python def sections( q:str, # query string emb:bytes, # query embedding limit:int=5, # sections to return per:int=3, # snippets kept per section store:str='store', prefix:str=None, fanout:int=8, # chunk hits gathered before rolling up score:str='max', # how a section scores from its hits: max | mean | sum rerank:bool=False, # reorder the chunk hits before they are grouped into sections **kw )->list: # forwarded to doc_search ``` *Ranked* sections*, not chunks: hits grouped by node, each with a `read` handle.* ------------------------------------------------------------------------ source ### Database.doc_search ``` python def doc_search( q:str, # query string emb:bytes, # query embedding columns:list=None, # extra columns (node_id, page, heading, doc_id always included) limit:int=10, store:str='store', prefix:str=None, spans:bool=True, # merge adjacent hits inside a node gap:int=1, # span merge tolerance, in pages adaptive:bool=False, # weight the RRF legs by query shape (measured inert — see below) rrf_k:int=60, rerank:bool=False, # reorder the candidates with a flashrank cross-encoder **kw )->list: # forwarded to Database.search ``` *Hybrid search over a node-aware store: span merging and a breadcrumb per hit.* ------------------------------------------------------------------------ source ### merge_spans ``` python def merge_spans( hits, # ranked hits carrying node_id + page gap:int=1, # merge hits at most this many pages apart )->list: ``` *Collapse hits that are adjacent inside the same node into one span.* ------------------------------------------------------------------------ source ### adaptive_weights ``` python def adaptive_weights( q:str, # the raw query fts:list, # the FTS leg's results vec:list, # the vector leg's results )->tuple: ``` *`(fts_weight, vec_weight)` for RRF, from cheap signals in the query and the legs.* `context` composes one retrieval: the operative sections plus what they connect to. `ann`, `rerank` and `fts_pre` reach [`Database.search`](https://Karthik777.github.io/litesearch/core.html#database.search) through it. ------------------------------------------------------------------------ source ### Database.context ``` python def context( q:str, # query string emb:bytes, # query embedding store:str='store', prefix:str=None, sections:int=6, # operative sections returned per:int=3, # snippets kept per section graph:bool=False, # opt in to graph-reached related sections (see the note below) vector:bool=True, # include embedding-nearest related sections related:int=8, # max related sections max_read:int=6000, # chars of assembled text per operative section tree_ctx:bool=True, # attach each section's parent/siblings/children graph_w:float=0.6, **kw ): ``` *One composed retrieval over a document tree: the operative sections plus what they connect to.* ------------------------------------------------------------------------ source ### Database.node_context ``` python def node_context( node_id:str, # a 'doc#seq' node id store:str='store', prefix:str=None ): ``` *Where a section sits: its parent, siblings and children — a hit as a provision in a structure.* ## End to end A three-chapter document, a hash embedder so the notebook runs offline, and every call in the module. ``` python import numpy as np, hashlib from litesearch.core import database from litesearch.utils import hash_embed DOC = """# Saturn Saturn is the slowest of the classical grahas. ## Sade Sati Sade sati is the seven and a half year period when Saturn transits the twelfth, first and second houses from the natal moon. It is counted in three phases of roughly thirty months each. The middle phase, with Saturn over the moon itself, is held to be the heaviest. ## Remedies Recitation on Saturdays is the common prescription. Donation of black sesame is another. # Jupiter Jupiter is the great benefic and moves through one sign each year. ## Transits Jupiter's return to its natal sign happens roughly every twelve years.""" db = database() enc = lambda ts, **kw: hash_embed(ts, 256) db.get_tree('store') print(db.add_doc(DOC, title='Grahas', source='notes/grahas.md', kind='md', emb_fn=enc)) ``` {'doc_id': '5a991f283775ae0d', 'title': 'Grahas', 'kind': 'md', 'nodes': 6, 'chunks': 5} ``` python # the tree came out of the headings, and every chunk knows which node it lives in t = db.toc('Grahas')[0] def show(n, d=0): print(' '*d + f"{n['title']:<14} chunks={n['nchunks']} {n['id']}") for c in n.get('children', []): show(c, d+1) show(t['tree']) assert [c['title'] for c in t['tree']['children']] == ['Saturn', 'Jupiter'] assert db.breadcrumb(t['tree']['children'][0]['children'][0]['id']) == 'Grahas › Saturn › Sade Sati' ``` Grahas chunks=0 5a991f283775ae0d#0 Saturn chunks=1 5a991f283775ae0d#1 Sade Sati chunks=1 5a991f283775ae0d#2 Remedies chunks=1 5a991f283775ae0d#3 Jupiter chunks=1 5a991f283775ae0d#4 Transits chunks=1 5a991f283775ae0d#5 ``` python # read() returns the section, not a fragment -- and includes its children sec = first(t['tree']['children'][0]['children'], lambda c: c['title'] == 'Sade Sati') r = db.read(sec['id']) print(r['breadcrumb'], '|', r['pages'], '|', len(r['text']), 'chars') assert 'thirty months' in r['text'] and r['breadcrumb'].endswith('Sade Sati') ``` Grahas › Saturn › Sade Sati | (0, 0) | 266 chars ``` python # hybrid search, but every hit is placed in the document q = 'how long does the saturn transit last' hits = db.doc_search(q, enc([q])[0].tobytes(), limit=3, dtype=np.float16) for h in hits: print(f"{h['_rrf_score']:.4f} {h['breadcrumb']:<28} {h['content'][:60]!r}") assert hits and all(h['breadcrumb'] for h in hits) # sections() rolls the same evidence up a level and hands back the next call secs = db.sections(q, enc([q])[0].tobytes(), limit=2, dtype=np.float16) for s in secs: print(f"{s['score']:.4f} {s['breadcrumb']:<28} {s['read']}") assert secs and secs[0]['read'].startswith('read(') ``` 0.0333 Grahas › Saturn › Sade Sati 'Sade sati is the seven and a half year period when Saturn tr' 0.0328 Grahas › Saturn 'Saturn is the slowest of the classical grahas.' 0.0161 Grahas › Saturn › Remedies 'Recitation on Saturdays is the common prescription. Donation' 0.0333 Grahas › Saturn › Sade Sati read('5a991f283775ae0d#2') 0.0328 Grahas › Saturn read('5a991f283775ae0d#1') ``` python _rq = 'how long does the saturn transit last' _rqv = enc([_rq])[0].tobytes() _plain = db.doc_search(_rq, _rqv, limit=3, dtype=np.float16) _rr = db.doc_search(_rq, _rqv, limit=3, dtype=np.float16, rerank=True) assert len(_rr) <= 3 and all(h['breadcrumb'] for h in _rr) assert [h['_rrf_score'] for h in _rr] == [1.0/(60+i) for i in range(len(_rr))] assert [h['_rrf_score'] for h in _plain] != [1.0/(60+i) for i in range(len(_plain))] # and the roll-up inherits it: sections forwards rerank to the chunk hits it groups _srr = db.sections(_rq, _rqv, limit=2, dtype=np.float16, rerank=True) assert _srr and all(s['breadcrumb'] for s in _srr) print('reranked:', [h['content'][:40] for h in _rr]) ``` INFO:flashrank.Ranker:Downloading ms-marco-TinyBERT-L-2-v2... ms-marco-TinyBERT-L-2-v2.zip: 0%| | 0.00/3.26M [00:00 adaptive_weights('how long does it last', [1], [1])[0] assert adaptive_weights('anything', [], [1]) == (0.0, 1.0) assert adaptive_weights('anything', [1], []) == (1.0, 0.0) # spans: two hits one page apart in the same node become one passage _h = [dict(content='first half', node_id='d#1', page=3, _rrf_score=0.02, rowid=1), dict(content='second half', node_id='d#1', page=4, _rrf_score=0.01, rowid=2), dict(content='elsewhere', node_id='d#9', page=40, _rrf_score=0.005, rowid=3)] _m = merge_spans(_h) assert len(_m) == 2 and _m[0]['_nspan'] == 2 and 'second half' in _m[0]['content'], _m assert _m[0]['_rrf_score'] == 0.02, 'a merged span keeps its best score, it does not accumulate' ``` ``` python # ingestion is content-addressed, and delete_doc leaves nothing behind again = db.add_doc(DOC, title='Grahas', source='notes/grahas.md', emb_fn=enc) assert 'skipped' in again, again did = t['doc_id'] n_before = len(db.t.store()) db.delete_doc(did) assert len(db.t.store()) == 0 and db.toc() == [] and n_before > 0 print(f'{n_before} chunks removed with the document') ``` 5 chunks removed with the document ## Not done yet - **No LLM.** `summarize=` takes a callable. A summary per node is what `toc()` shows an agent deciding where to read. - **Chunking is [`chunk_markdown`](https://Karthik777.github.io/litesearch/data.html#chunk_markdown).** A cost-model splitter produces better boundaries on prose. The seam is `chunker=`. - **Chunking is where the gains are.** In `07_doc_eval`, node-scoped chunks lifted MRR from 0.330 to 0.522 on degraded queries, more than everything else here combined. ``` python import tempfile _d = Path(tempfile.mkdtemp()) (_d/'ratios.md').write_text('# Ratios\n\n## Liquidity\n\nCurrent ratio is current assets over current liabilities.') (_d/'notes.txt').write_text('A plain text file with no headings at all, ingested as one windowed node.') _fdb = database() _fdb.get_tree('store') print(_fdb.add_dir(_d, emb_fn=enc)) assert {d['title'] for d in _fdb.toc()} == {'ratios', 'notes'} _rt = _fdb.toc('ratios')[0]['tree'] assert [n['title'] for n in _rt['children']] == ['Ratios'] # the `# Ratios` heading assert [n['title'] for n in _rt['children'][0]['children']] == ['Liquidity'] assert _fdb.add_file(_d/'ratios.md', emb_fn=enc).get('skipped'), 're-ingest is a no-op' assert 'current liabilities' in _fdb.read(_fdb.toc('ratios')[0]['tree']['id'])['text'] ``` [{'doc_id': '1e8d58569295fb93', 'title': 'notes', 'kind': 'txt', 'nodes': 2, 'chunks': 1}, {'doc_id': 'b4432a03f1d158da', 'title': 'ratios', 'kind': 'md', 'nodes': 3, 'chunks': 1}]