tree

Document structure, so search can return whole sections with a breadcrumb

What a tree adds

A plain chunk store can tell you which 512-byte piece of text matches a query. It cannot tell you which chapter or section that piece belongs to, and a piece cut from the middle of a section often makes little sense on its own. This module records the structure of each document so search results can be placed and read in context.

At ingest, each document is split into a tree of sections. A node is one section: the document itself is the root node, and each heading (or chapter line, or citation, or group of pages) opens a child node below it. Every node has a node_id of the form <doc id>#<seq>, where seq is the node’s position in the document. Every chunk is stored with the node_id of the section it came from and its heading path, the chain of section titles from the document down to that section, such as Grahas › Saturn › Sade Sati. Results call this path the breadcrumb.

With the tree in place you get:

• doc_search, hybrid search whose hits carry a breadcrumb and whose neighbouring hits in one section are joined;
• sections, search that returns the best sections instead of chunks;
• toc and read, which list a document’s sections and return one whole section, without computing any embeddings;
• context, which gathers the best sections for a question, with their full text, for a language model to read.

Nothing here calls a language model. Node summaries are the first 300 characters of the section unless you pass your own function as summarize=.

Index (see the api page) calls all of these for you with its encoder already chosen. Call them directly when you build your own database().

Detecting structure

build_tree needs to know what marks a new section in a document. detect_mode looks for four kinds of signal and picks one mode per document:

mode	sections start at	typical source
markdown	# headings	markdown files, PDFs converted to markdown
verse	verse citations such as || Mn_1.1 ||, and # headings	Sanskrit texts from GRETIL or TEI
chapter	lines like CHAPTER IV, Article 12 or ADHYAYA 3	scanned or plain-text books and legislation
window	every 8 pages	anything with none of the above

The window mode is the fallback, so every document gets a tree of some kind. A document counts as markdown with at least 2 headings if it has up to 3 pages, and otherwise at least 3 headings or one per 25 pages, whichever is more, so a one-page note with a single heading becomes one window. Some PDF converters mark every bold line as a top-level heading. When a document has more than 4 headings per page, all at the same depth, detect_mode does not trust them and uses chapter lines or windows instead.

In chapter mode, struct_levels decides how the structural words nest. Known words have a fixed order: BOOK, TITLE, PART and similar come first, then CHAPTER, then SECTION, then ARTICLE, RULE and CLAUSE. Levels are then renumbered from 1 with no gaps, so a document that only uses “Article” gets its articles at level 1. At most 4 levels are kept.

detect_mode checks for verse mode first. It chooses verse when at least 3 pages carry a citation, citations appear on more than a fifth of the pages, they name at least two levels (such as chapter and verse), and they outnumber markdown headings.

---

source

detect_mode

def detect_mode(
    pages
)->str:

Choose how to find the sections of a document: markdown, verse, chapter or window.

---

source

struct_levels

def struct_levels(
    pages, max_levels:int=4
)->dict:

Map each structural word in a document, such as CHAPTER or ARTICLE, to a nesting level counted from 1.

---

source

summarize_extractive

def summarize_extractive(
    text:str, n:int=300
)->str:

Return the first n characters of text with whitespace collapsed, the default node summary.

---

source

TreeNode

def TreeNode(
    seq:int, title:str, level:int, parent:int | None=None, page_start:int=0, page_end:int=0, segments:list=<factory>,
    children:list=<factory>, summary:str=''
)->None:

One section of a document, where seq is its position in the node list and parent is the seq of the section above it.

build_tree takes a list of (page number, text) pairs and returns the nodes as a flat list, where nodes[seq] is the node with that seq and each node’s parent is its parent’s seq. The same list is written to the nodes table, and toc() rebuilds the nesting when you ask for it. A heading is placed under the nearest earlier heading of a shallower level, so a document that starts at ## or jumps from # to ### still nests correctly.

In verse mode, a citation like Mn_1.1 gives the verse’s address: every part except the last names the division it belongs to, so Mn_1.1 opens a node Mn 1 for chapter 1. Verse mode also reads # headings, because some texts name their sections (## dhyanam) and carry no citations. Citation levels are counted from the heading above them.

heading_path builds the breadcrumb for a node. If a segment repeats the one before it, it is dropped, so a PDF whose first heading is the same as the document title reads Field Manual › Chapter 1 and not Field Manual › Field Manual › Chapter 1.

The breadcrumb is stored with every chunk in the heading column, and by default it is also prepended to the chunk’s text before embedding, so the vector reflects where the text sits. It is not added to the full-text index: a keyword search for a section title finds only chunks whose own text contains it.

---

source

heading_path

def heading_path(
    tree, # the node list from build_tree
    nd, # the TreeNode to describe
    title:str, # document title
    sep:str=' › ', # separator between titles
    max_len:int=200
)->str: # longest breadcrumb kept, in characters

Return the breadcrumb of one node, such as Doc › Part › Chapter.

---

source

build_tree

def build_tree(
    pages, # list of (page number, text); the text may be markdown or plain
    title:str='Document', # title of the root node
    summarize:NoneType=None, # function from section text to a summary; None keeps the first 300 characters
    window:int=8, # pages per node in `window` mode
    min_level:int=1, # floor for heading depth
    max_levels:int=4, # deepest node level kept
    mode:str=None, # `markdown`, `verse`, `chapter` or `window`; None detects it
)->list:

Split a document into sections and return them as a flat list of TreeNode, with the root at index 0.

tree = build_tree([(0, '# Saturn\n\nIntro text about the ringed planet.\n\n## Transits\n\nSade sati runs seven years.'),
                   (1, '## Remedies\n\nRecite on Saturdays.')], title='Jyotisha')
for n in tree: print(f'{n.seq}  lvl{n.level}  p{n.page_start}-{n.page_end}  {n.title!r:24} {n.summary[:40]!r}')
assert [n.title for n in tree] == ['Jyotisha', 'Saturn', 'Transits', 'Remedies']
assert tree[2].parent == 1 and tree[3].parent == 1, 'h2s hang off the h1, not off each other'
assert heading_path(tree, tree[2], 'Jyotisha') == 'Jyotisha › Saturn › Transits'

0  lvl0  p0-1  'Jyotisha'               'Saturn'
1  lvl1  p0-0  'Saturn'                 'Intro text about the ringed planet.'
2  lvl2  p0-0  'Transits'               'Sade sati runs seven years.'
3  lvl2  p1-1  'Remedies'               'Recite on Saturdays.'

# a heading is a heading whatever its first letter: `## verses 6` used to be dropped, shifting the nodes after it
_low = build_tree([(0, '# Book\n\nintro\n\n## verses 1\n\nalpha\n\n## Verses 2\n\nbeta\n\n## verses 6\n\ngamma')], title='Doc')
assert [n.title for n in _low] == ['Doc', 'Book', 'verses 1', 'Verses 2', 'verses 6'], [n.title for n in _low]
assert [n.parent for n in _low] == [None, 0, 1, 1, 1]
assert 'gamma' in _low[4].text() and 'alpha' in _low[2].text()

# a skipped heading level must not nest siblings under each other. Both of these are ordinary
# documents: a README with no h1, so `##` is the shallowest heading, and a document that jumps
# from `#` straight to `###`.
_gap = build_tree([(0, '> a tagline\n\n## Install\n\npip install\n\n'
                      '## Encoders\n\nencoders\n\n### Fallback\n\nhashing')], title='litesearch')
assert [n.title for n in _gap] == ['litesearch', 'Install', 'Encoders', 'Fallback']
assert [n.parent for n in _gap] == [None, 0, 0, 2], [n.parent for n in _gap]
assert heading_path(_gap, _gap[2], 'litesearch') == 'litesearch › Encoders'
assert heading_path(_gap, _gap[3], 'litesearch') == 'litesearch › Encoders › Fallback'

_jump = build_tree([(0, '# Doc\n\ntop\n\n### Deep One\n\na\n\n### Deep Two\n\nb')], title='Doc')
assert [n.parent for n in _jump] == [None, 0, 1, 1], [(n.title, n.level, n.parent) for n in _jump]

# a `#` comment inside a fenced block is a comment, not an h1 — otherwise a README's usage examples
# invent sections that adopt every real heading after them
_readme = '''# vishalakshi

> a tagline

## Harvest

```python
v.apis('https://shop/browse')
# [0] .../api/bff/products?page=1   records: 24
```

## Watches

watch what changes
'''
_fen = build_tree([(0, _readme)], title='vishalakshi')
assert [n.title for n in _fen] == ['vishalakshi', 'vishalakshi', 'Harvest', 'Watches'], [n.title for n in _fen]
assert [n.parent for n in _fen] == [None, 0, 1, 1]
assert 'records: 24' in _fen[2].text(), 'the fenced lines are still text, only never headings'
assert _md_stats([(0, _readme)]) == (3, 2)        # h1 + 2 h2; the comment is not a heading
assert detect_mode([(0, _readme)]) == 'markdown'

# an unclosed fence swallows the rest of the document rather than mis-reading it as structure
_open = build_tree([(0, '# Doc\n\n## Alpha\n\n```\n## Beta\n')], title='Doc')
assert [n.title for n in _open] == ['Doc', 'Doc', 'Alpha'], [n.title for n in _open]
assert '## Beta' in _open[2].text()

# a ````-fenced block quotes ``` blocks inside itself: only a fence at least as long closes it
_nest = '# Doc\n\n## Alpha\n\n````md\n```\n# Not a heading\n```\n````\n\n## Beta\n\nb'
assert [n.title for n in build_tree([(0, _nest)], title='Doc')] == ['Doc', 'Doc', 'Alpha', 'Beta']
# and the two fence characters do not close each other
assert list(_md_lines('~~~\n# x\n```\n# y\n~~~')) == [
    ('~~~', True), ('# x', True), ('```', True), ('# y', True), ('~~~', True)]

# a scanned book has chapter lines, not markdown; and a doc with neither still gets a tree
_book = build_tree([(i, f'CHAPTER {i+1}\n\nbody of chapter {i+1}') for i in range(4)], title='Saravali')
assert detect_mode([(i, f'CHAPTER {i+1}\n\nbody') for i in range(4)]) == 'chapter'
assert len(_book) == 5 and _book[1].title.startswith('CHAPTER 1')

_plain = build_tree([(i, f'page {i} of undifferentiated prose') for i in range(20)], title='Notes', window=8)
assert detect_mode([(i, 'prose') for i in range(20)]) == 'window'
assert len(_plain) == 4, [n.title for n in _plain]   # root + ceil(20/8) windows
assert _plain[1].title.startswith('Pages 1–8')

The tables

get_tree adds three tables beside a chunk store and four columns to the store. The chunk store is still an ordinary store, so full-text search, the ANN index and db.search work on it unchanged.

docs    id · title · source · kind · pages · meta · hash · added_at
sources source · doc_id · added_at
nodes   id ('doc#seq') · doc_id · parent_id · level · seq · title ·
        page_start · page_end · summary · nchunks
store   content · embedding · metadata · doc_id · node_id · page · heading   [+ FTS5, +ANN]

For the default store the tables are called docs and nodes; for a store called papers they are papers_docs, papers_sources and papers_nodes.

A document’s id is a hash of its source and its title. A file’s source is its resolved path, so a relative path, an absolute path and a symlink to one file are one source. hash is a hash of the document’s (page, text) pairs, and sources maps every source to the document holding its text:

• The same source with the same text again is skipped as unchanged.
• The same source with changed text replaces the old chunks and nodes, and keeps the id.
• A different source with text already stored is skipped as same text as <source>, and gains a row in sources pointing at the stored document. A copy, or a string with the same text, adds no chunks.
• A document goes when the last source pointing at it does. delete_source forgets one source. delete_doc forgets the document’s own source; text another source shares stays, under that source. Editing a file whose text another source shares moves the old text to an id of its own for that source first.
• A text added with no source is keyed by its title. A second, different text under the same title is skipped, as before 0.1.50, because a title names nothing on disk.

force=True rebuilds a document even when its text is stored, which is how a new chunker, mode or meta_fn reaches text already ingested. A title or kind change on one source builds a new document under the new id and drops the old one.

A store written before 0.1.50 has no sources table and no hashes. get_tree creates the table and fills it from docs.source. Re-adding a file by the path it was first added under moves its document to the resolved-path id, keeping its vectors, and re-embeds it once to record its hash. Duplicates an old store already holds go as their sources are re-added.

---

source

Database.get_tree

def get_tree(
    store:str='store', # chunk store the tree is built over
    prefix:str=None, # table prefix (default: '' for 'store', else '<store>_')
    ann:bool=True, # register an ANN index on the chunk store
    encoder:str=None, # the encoder whose vectors the chunk store holds; checked on reopen
    **kw
)->fastcore.basics.AttrDict: # extra typed columns for the chunk store

Create the docs, nodes and sources tables and a chunk store with node columns, or open them if they exist.

---

source

source_key

def source_key(
    s
)->str:

Return the key a source is stored under: the resolved absolute path for an existing path, else the string.

---

source

is_path

def is_path(
    s
)->bool:

Return whether s names an existing file or folder; a string too long to be a path is not one.

---

source

doc_hash

def doc_hash(
    pages
)->str:

Return a hash of a document’s (page, text) pairs, the same from any source.

---

source

doc_id

def doc_id(
    source, title:str=''
)->str:

Return a 16-character document id computed from the source and title.

Ingestion

There are three ways in, from the most control to the least:

• add_doc takes text you have already extracted, as a string or a list of (page number, text) pairs. It builds the tree, chunks each section, embeds the chunks with emb_fn and stores them. Use it when your app fetches or converts documents itself.
• add_file reads one file and passes its pages to add_doc.
• add_dir does the same for every document file under a folder, or for a list of files.

Each returns a dict per document with its doc_id, title, kind, number of nodes and number of chunks, or a skipped reason when its text is already stored: unchanged, or same text as <source>.

emb_fn takes a list of strings and returns one vector per string. Without it, chunks are stored with no embedding, which is enough for keyword search, toc and read.

Chunks are cut within each section, never across a section boundary, using chunk_markdown’s default chunker unless you pass chunker=, any chonkie chunker. A chunk shorter than 40 characters is joined to the chunk before it instead of being stored alone. Two chunks with the same text in different sections are both kept, because the row id is a hash of the node id and the text.

A chunker sees one segment of a section at a time. Some readers produce very small segments, such as one verse per page, and a chunker aiming at 700 characters never reaches that size on a 60-character verse. A chunker can ask for its small chunks to be joined across segments by having pack_cited=True and a target size; pack_target reads the size a chunker asks for, and _pack_segments then joins consecutive chunks of one node up to target characters. Ordinary chunkers have neither attribute and are unaffected.

add_doc(defer=...) collects the chunks in a list instead of embedding and storing them, and store_chunks embeds and stores such a list later. add_doc(jobs=...) collects the whole write, and _commit stores a batch of them: chunks first, then nodes, doc rows and sources. add_dir uses this to embed chunks from many documents in batches of 2,000 (embed_batch), which is faster than one call per document, and a run that fails before a batch is stored leaves the documents it would have replaced whole.

---

source

Database.delete_source

def delete_source(
    source:str, store:str='store', prefix:str=None
)->dict:

Forget one source; its document goes too unless another source still points at it.

---

source

Database.delete_doc

def delete_doc(
    did:str, store:str='store', prefix:str=None
):

Remove a document with its nodes and chunks; text another source shares stays, under that source.

---

source

store_chunks

def store_chunks(
    store, # chunk store table
    chunks, # chunk dicts from `_node_chunks`, from one document or many
    emb_fn:NoneType=None, # function from a list of strings to vectors
    with_heading:bool=True, # embed each chunk together with its heading path
):

Embed a batch of chunks, which may come from several documents, and upsert them into the store.

---

source

Database.add_doc

def add_doc(
    pages, # list of (page number, text), or one string
    title:str, # document title
    source:str=None, # path or url; an existing path is stored resolved. None keys the text by its title
    kind:str='text', # a label stored on the doc row to filter on, such as 'pdf', 'web' or 'md'
    store:str='store', # chunk store
    prefix:str=None, # tree table prefix
    emb_fn:NoneType=None, # function from a list of strings to vectors; None stores no embeddings
    chunker:NoneType=None, # chonkie chunker; None uses chunk_markdown's default
    summarize:NoneType=None, # function from section text to a summary
    with_heading:bool=True, # embed each chunk together with its heading path
    meta:dict=None, # arbitrary json metadata for the doc row
    force:bool=False, # rebuild the document even when its text is already stored
    index:bool=True, # mirror this document's chunks into the ANN index
    mode:str=None, # force a build_tree structural mode instead of detecting one
    meta_fn:NoneType=None, # (chunk text) -> dict of facets stored as the chunk's `metadata`
    defer:list=None, # collect chunks here instead of embedding and storing them
    jobs:list=None, # collect the whole write here for `_commit`, as `add_dir` does
)->dict:

Ingest one document: build its tree, chunk each section, embed the chunks and store them.

---

source

pack_target

def pack_target(
    chunker
)->int:

Return the size a chunker asks consecutive segments to be joined up to, or 0 to chunk each segment on its own.

add_file handles document files: PDF, markdown, plain text, reStructuredText, notebooks, XML, HTML and CoNLL-U. XML, HTML and CoNLL-U are read as plain text unless a registered profile claims them. A PDF is converted to one markdown page per PDF page by pdf_parse, and its images are saved under db.assets(), a folder beside the database file. A notebook becomes one page with its cells joined. Any other file is read as one page of text. The title defaults to the file name with underscores and hyphens turned into spaces, and kind defaults to the extension (notebook for .ipynb).

Source code does not fit a heading-based tree. To index code, use Index.add_code (see the api page), which cuts it into top-level functions and classes using Python’s syntax tree.

A Profile (see the data page) describes how to read one kind of document: its parser, chunker, tree mode and per-chunk metadata. When a registered profile claims a file, add_file uses it instead of the steps above. This is how a TEI edition of a Sanskrit text is told apart from any other .xml file. Pass profile= with a name to choose a profile directly, which is the only way to reach one that cannot be detected from the file.

---

source

Database.add_file

def add_file(
    path, # file to ingest
    title:str=None, # None uses the file name, resolved through symlinks, with underscores and hyphens as spaces
    store:str='store', prefix:str=None, kind:str=None, # overrides the kind inferred from the extension
    out_path:NoneType=None, # dir for extracted PDF images; defaults to `assets/<stem>` beside the db
    profile:str=None, # force a registered Profile by name instead of detecting one
    **kw
)->dict: # forwarded to add_doc (emb_fn, chunker, summarize, force, ...)

Ingest one document file, read by a matching Profile or by its extension.

---

source

Database.assets

def assets(
    name:str=None
)->pathlib.Path:

Return the folder for extracted assets such as PDF images, beside the database file or in the temp folder for an in-memory database.

from fastcore.test import test_eq
test_eq(_n_parse_workers([Path(f'p{i}.html') for i in range(8)], None), 0)
test_eq(_n_parse_workers([Path(f'p{i}.pdf') for i in range(4)], None), defaults.cpus)
test_eq(_n_parse_workers([Path('a.html')], 3), 3)

add_dir ingests every file under dir whose extension is in types, or exactly the paths in files=. Files whose text is already stored are skipped, and a file edited since it was added is replaced. Each call writes to one store, because the embedder and the ANN index belong to that store; to send files from one folder to different stores, call it once per store with files=.

Parsing PDFs, notebooks and XML is slow, so when 4 or more such files are in the batch, add_dir parses them in a pool of worker processes (n_workers overrides this; 0 means no pool). In one measurement, parsing took 3.55 of the 5.31 seconds needed to ingest 8 PDFs, while 150 markdown files took 2.7 milliseconds to read in total, too little for a pool to help, and HTML is read the same way. All writes stay in the main process.

By default the full-text index is updated as each chunk is written. With bulk=True, add_dir turns that off during the load and rebuilds the full-text index once at the end. This is faster for a large first load, but keyword search misses the new documents until the load finishes, so use it only when nothing searches the store meanwhile. The ANN index is rebuilt once at the end either way.

stop is a callable add_dir asks before each file, for a caller that may need to cancel a long load. When it returns True, add_dir stops: the files already stored stay stored and indexed, a parse in flight finishes, and no further parse starts. Given stop, the result is an L whose cancelled says whether files went unread and whose unread lists them. Without stop, the result is the plain list it always was.

---

source

Database.add_dir

def add_dir(
    dir:NoneType=None, # directory to walk; ignored when `files` is given
    types:str='.pdf,.md,.markdown,.txt,.rst,.ipynb,.xml,.tei,.htm,.html,.conllu', # comma-separated extensions to ingest
    store:str='store', prefix:str=None,
    n_workers:int=None, # parse workers; 0 is serial, None picks by parse-heavy file count
    embed_batch:int=2000, # chunks embedded and written per flush; 0 writes per document
    files:NoneType=None, # ingest exactly these paths instead of walking `dir`
    bulk:bool=False, # rebuild the full-text index once at the end instead of per chunk
    stop:NoneType=None, # asked before each file; True stops, and the result's `unread` holds the rest
    **kw
)->list: # forwarded to add_doc

Ingest every document file under a folder, or a given list of files, skipping those already ingested; given stop, the result is an L carrying cancelled and unread.

from tempfile import TemporaryDirectory
from litesearch.core import database
with TemporaryDirectory() as d:
    Path(d, 'a.md').write_text('# Alpha\n\npolonium report')
    x = database(Path(d)/'test.db', sem_search=False)
    x.add_dir(d, types='md')
    assert x.t.store.fts_search('polonium')

with TemporaryDirectory() as d:
    Path(d, 'a.md').write_text('# Alpha\n\npolonium report')
    Path(d, 'b.md').write_text('# Beta\n\nthorium report')

    x = database(Path(d)/'test.db', sem_search=False)
    x.add_dir(d, types='md', bulk=True)

    assert len(x.t.store.fts_search('polonium')) == 1
    assert len(x.t.store.fts_search('thorium')) == 1

    # Triggers were restored after the rebuild.
    x.t.store.insert({'content': 'later radium report'})
    assert len(x.t.store.fts_search('radium')) == 1

from fastcore.test import test_eq
with TemporaryDirectory() as d:
    for n in 'abcd': Path(d, f'{n}.md').write_text(f'# {n}\n\n{n} report')
    x, asked = database(Path(d)/'test.db', sem_search=False), []
    def stop(): asked.append(1); return len(asked) > 2
    r = x.add_dir(d, types='md', stop=stop)
    test_eq([o['title'] for o in r], ['a', 'b'])
    assert r.cancelled and [p.name for p in r.unread] == ['c.md', 'd.md']
    assert len(x.t.store.fts_search('report')) == 2, 'what was stored before the stop stays searchable'
    r = x.add_dir(d, types='md', stop=lambda: False)
    assert not r.cancelled and not r.unread
    test_eq(sorted(o['title'] for o in r), ['a', 'b', 'c', 'd'])
    assert type(x.add_dir(d, types='md')) is list, 'no stop, no change'
    y, asked = database(Path(d)/'pool.db', sem_search=False), []
    r = y.add_dir(d, types='md', n_workers=2, stop=stop)
    assert r.cancelled and len(r) == 2 and len(r.unread) == 2, 'the pool path stops between files too'
    r = y.add_dir(files=[], stop=lambda: True)
    assert not r.cancelled and not r.unread and not r

Reading the structure

These functions read the tree and compute no embeddings, so they are fast and need no encoder. They suit an agent or a reader that browses: look at the table of contents, pick “the chapter on remedies”, and open it.

• toc(doc) returns each document’s tree as nested dicts with node ids, titles, levels, page ranges, chunk counts and summaries. doc may be a doc id or part of a title; leave it out for every document. max_depth (3) limits how deep it goes.
• read(node_id) returns one whole section: its chunks joined in order, with the text of all its subsections by default (children=False for just the section), up to max_chars (20,000). It also returns the breadcrumb, page range, summary and the ids of its direct children.
• breadcrumb(node_id) returns just the title path.

---

source

Database.read

def read(
    node_id:str, # 'doc#seq', as returned by toc() or sections()
    store:str='store', prefix:str=None, max_chars:int=20000, # cap on the assembled text
    children:bool=True, # append the text of child nodes too
)->dict:

Return the full text of one section, with its subsections by default, plus its breadcrumb, pages and summary.

---

source

Database.breadcrumb

def breadcrumb(
    node_id:str, store:str='store', prefix:str=None, sep:str=' › '
)->str:

Return the titles from the document down to a node, such as Book › Chapter › Section.

---

source

Database.toc

def toc(
    doc:str=None, # doc id, or part of a title; None lists every document
    store:str='store', prefix:str=None, max_depth:int=3, # deepest level included
    summaries:bool=True, # include node summaries
)->list:

Return the document tree as nested dicts of titles, page ranges, chunk counts and summaries.

Retrieval that knows about structure

Hybrid search runs two searches and combines them. The keyword search uses SQLite FTS5, a full-text index ranked by bm25, which favours chunks that contain the query’s rarer words. The vector search compares the query’s embedding with each chunk’s embedding. Reciprocal Rank Fusion (RRF) merges the two ranked lists: each chunk scores 1/(rrf_k + rank) in each list it appears in, so a chunk ranked well by both searches beats one ranked well by only one. rrf_k defaults to 60.

doc_search is db.search for a tree store, with two additions. Every hit gets a breadcrumb. And hits from the same section at most gap pages apart (1 by default) are joined into one hit, so a reader gets one passage instead of two halves; the joined hit keeps the best rank and score of its parts and counts them in _nspan. rerank=True reorders the top 30 candidates with a flashrank cross-encoder, a small model that reads the query and each chunk together; it is slower and downloads a 3 MB model on first use.

sections answers “which sections match”, returning sections instead of chunks. It gathers chunk hits (limit × fanout, at least 20), groups them by node, and scores each section by its best hit (score='max'). Each section comes with its breadcrumb, summary, page range, up to per snippets and a read string naming the call that opens it. score='mean' averages the hits and score='sum' adds them. Summing favours long sections that collect many weak hits: on 150 known-item queries over 486 pages of EU legislation (07_doc_eval), sections scored 0.852 MRR with sum against 0.957 with max on verbatim queries, and 0.304 against 0.505 on reworded ones.

Use doc_search when you want passages to show or quote. Use sections when the answer is a place in a document, such as which article or chapter, and you will read it next.

doc_search(adaptive=True) changes the weight of each list before fusion using adaptive_weights, three fixed rules: when one search returns nothing, the other gets all the weight; a quoted phrase adds 0.6 to the keyword weight; and a query where at least half the words look like names or identifiers (capitalised, containing digits or underscores) adds 0.4. It is off by default because on the natural-language queries in 07_doc_eval the rules never fired.

---

source

Database.sections

def sections(
    q:str, # query string
    emb:bytes, # query embedding
    limit:int=5, # sections to return
    per:int=3, # snippets kept per section
    store:str='store', prefix:str=None, fanout:int=8, # chunk hits gathered before rolling up
    score:str='max', # how a section scores from its hits: max | mean | sum
    rerank:bool=False, # reorder the chunk hits before they are grouped into sections
    **kw
)->list: # forwarded to doc_search

Return the best-matching sections for a query, each with snippets and a read call that opens it.

---

source

Database.doc_search

def doc_search(
    q:str, # query string
    emb:bytes, # query embedding
    columns:list=None, # extra columns (node_id, page, heading, doc_id always included; each hit also gets its document's `source`)
    limit:int=10, store:str='store', prefix:str=None, spans:bool=True, # merge adjacent hits inside a node
    gap:int=1, # span merge tolerance, in pages
    adaptive:bool=False, # weight the two result lists by query shape (see `adaptive_weights`)
    rrf_k:int=60, # RRF constant; a larger value narrows the gap between ranks
    rerank:bool=False, # reorder the candidates with a flashrank cross-encoder
    **kw
)->list: # forwarded to Database.search

Hybrid search over a tree store that joins neighbouring hits in a section and gives every hit a breadcrumb.

---

source

merge_spans

def merge_spans(
    hits, # ranked hits carrying node_id + page
    gap:int=1, # merge hits at most this many pages apart
)->list:

Collapse hits that are adjacent inside the same node into one span.

---

source

adaptive_weights

def adaptive_weights(
    q:str, # the raw query
    fts:list, # the FTS leg's results
    vec:list, # the vector leg's results
)->tuple:

Return (fts_weight, vec_weight) for fusion, from the shape of the query and whether either result list is empty.

context gathers what a language model needs to answer a question from a document collection, in one call. It returns AttrDict(query, results, related):

• results holds up to sections (6) of the best sections. Each has its full text from read (up to max_read, 6,000 characters), breadcrumb, pages, summary, score, snippets, source file name, and with tree_ctx=True its place in the tree from node_context: its parent, up to 8 sibling sections and up to 12 child sections. A document’s root node is never returned as a result, and two sections with the same breadcrumb are returned once.
• related holds up to related (8) other sections that came up in a second hybrid search, with their title and breadcrumb but no text. Set vector=False to skip it.

Extra keyword arguments such as ann, rerank, fts_pre and rrf_k are passed to every search it runs.

---

source

Database.context

def context(
    q:str, # query string
    emb:bytes, # query embedding
    store:str='store', prefix:str=None, sections:int=6, # operative sections returned
    per:int=3, # snippets kept per section
    vector:bool=True, # also return related sections from a second hybrid search
    related:int=8, # max related sections
    max_read:int=6000, # chars of assembled text per operative section
    tree_ctx:bool=True, # attach each section's parent/siblings/children
    **kw
):

Return the best sections for a query with their full text and tree position, plus related sections.

---

source

Database.node_context

def node_context(
    node_id:str, # a 'doc#seq' node id
    store:str='store', prefix:str=None
):

Return a section’s parent, up to 8 siblings and up to 12 children.

End to end

The example below ingests a short document with two chapters, using hash_embed as the encoder so the notebook runs without downloading a model, and calls each function in turn.

import numpy as np, hashlib
from litesearch.core import database
from litesearch.utils import hash_embed

DOC = """# Saturn

Saturn is the slowest of the classical grahas.

## Sade Sati

Sade sati is the seven and a half year period when Saturn transits the twelfth, first and second
houses from the natal moon. It is counted in three phases of roughly thirty months each.

The middle phase, with Saturn over the moon itself, is held to be the heaviest.

## Remedies

Recitation on Saturdays is the common prescription. Donation of black sesame is another.

# Jupiter

Jupiter is the great benefic and moves through one sign each year.

## Transits

Jupiter's return to its natal sign happens roughly every twelve years."""

db  = database()
enc = lambda ts, **kw: hash_embed(ts, 256)
db.get_tree('store')
print(db.add_doc(DOC, title='Grahas', source='notes/grahas.md', kind='md', emb_fn=enc))

{'doc_id': '5a991f283775ae0d', 'title': 'Grahas', 'kind': 'md', 'nodes': 6, 'chunks': 5}

# the tree came out of the headings, and every chunk knows which node it lives in
t = db.toc('Grahas')[0]
def show(n, d=0):
    print('  '*d + f"{n['title']:<14} chunks={n['nchunks']}  {n['id']}")
    for c in n.get('children', []): show(c, d+1)
show(t['tree'])
assert [c['title'] for c in t['tree']['children']] == ['Saturn', 'Jupiter']
assert db.breadcrumb(t['tree']['children'][0]['children'][0]['id']) == 'Grahas › Saturn › Sade Sati'

Grahas         chunks=0  5a991f283775ae0d#0
  Saturn         chunks=1  5a991f283775ae0d#1
    Sade Sati      chunks=1  5a991f283775ae0d#2
    Remedies       chunks=1  5a991f283775ae0d#3
  Jupiter        chunks=1  5a991f283775ae0d#4
    Transits       chunks=1  5a991f283775ae0d#5

# read() returns the section, not a fragment -- and includes its children
sec = first(t['tree']['children'][0]['children'], lambda c: c['title'] == 'Sade Sati')
r = db.read(sec['id'])
print(r['breadcrumb'], '|', r['pages'], '|', len(r['text']), 'chars')
assert 'thirty months' in r['text'] and r['breadcrumb'].endswith('Sade Sati')

Grahas › Saturn › Sade Sati | (0, 0) | 266 chars

# hybrid search, but every hit is placed in the document
q  = 'how long does the saturn transit last'
hits = db.doc_search(q, enc([q])[0].tobytes(), limit=3, dtype=np.float16)
for h in hits: print(f"{h['_rrf_score']:.4f}  {h['breadcrumb']:<28} {h['content'][:60]!r}")
assert hits and all(h['breadcrumb'] for h in hits)

# sections() rolls the same evidence up a level and hands back the next call
secs = db.sections(q, enc([q])[0].tobytes(), limit=2, dtype=np.float16)
for s in secs: print(f"{s['score']:.4f}  {s['breadcrumb']:<28} {s['read']}")
assert secs and secs[0]['read'].startswith('read(')

0.0333  Grahas › Saturn › Sade Sati  'Sade sati is the seven and a half year period when Saturn tr'
0.0328  Grahas › Saturn              'Saturn is the slowest of the classical grahas.'
0.0161  Grahas › Saturn › Remedies   'Recitation on Saturdays is the common prescription. Donation'
0.0333  Grahas › Saturn › Sade Sati  read('5a991f283775ae0d#2')
0.0328  Grahas › Saturn              read('5a991f283775ae0d#1')

_rq  = 'how long does the saturn transit last'
_rqv = enc([_rq])[0].tobytes()
_plain = db.doc_search(_rq, _rqv, limit=3, dtype=np.float16)
_rr    = db.doc_search(_rq, _rqv, limit=3, dtype=np.float16, rerank=True)
assert len(_rr) <= 3 and all(h['breadcrumb'] for h in _rr)
assert [h['_rrf_score'] for h in _rr] == [1.0/(60+i) for i in range(len(_rr))]
assert [h['_rrf_score'] for h in _plain] != [1.0/(60+i) for i in range(len(_plain))]
# and the roll-up inherits it: sections forwards rerank to the chunk hits it groups
_srr = db.sections(_rq, _rqv, limit=2, dtype=np.float16, rerank=True)
assert _srr and all(s['breadcrumb'] for s in _srr)
print('reranked:', [h['content'][:40] for h in _rr])

INFO:flashrank.Ranker:Downloading ms-marco-TinyBERT-L-2-v2...

ms-marco-TinyBERT-L-2-v2.zip:   0%|          | 0.00/3.26M [00:00<?, ?iB/s]
ms-marco-TinyBERT-L-2-v2.zip:   1%|          | 24.0k/3.26M [00:00<00:17, 189kiB/s]
ms-marco-TinyBERT-L-2-v2.zip:   2%|▏         | 64.0k/3.26M [00:00<00:11, 304kiB/s]
ms-marco-TinyBERT-L-2-v2.zip:   4%|▍         | 136k/3.26M [00:00<00:06, 482kiB/s] 
ms-marco-TinyBERT-L-2-v2.zip:   8%|▊         | 272k/3.26M [00:00<00:03, 816kiB/s]
ms-marco-TinyBERT-L-2-v2.zip:  15%|█▌        | 512k/3.26M [00:00<00:02, 1.37MiB/s]
ms-marco-TinyBERT-L-2-v2.zip:  30%|██▉       | 992k/3.26M [00:00<00:00, 2.52MiB/s]
ms-marco-TinyBERT-L-2-v2.zip:  55%|█████▍    | 1.78M/3.26M [00:00<00:00, 4.42MiB/s]
ms-marco-TinyBERT-L-2-v2.zip: 100%|██████████| 3.26M/3.26M [00:00<00:00, 4.08MiB/s]

reranked: ['Sade sati is the seven and a half year p', 'Saturn is the slowest of the classical g', "Jupiter's return to its natal sign happe"]

# adaptive fusion: a quoted phrase leans on FTS, an empty FTS leg hands the vote to vectors
assert adaptive_weights('"sade sati"', [1], [1])[0] > adaptive_weights('how long does it last', [1], [1])[0]
assert adaptive_weights('anything', [], [1]) == (0.0, 1.0)
assert adaptive_weights('anything', [1], []) == (1.0, 0.0)

# spans: two hits one page apart in the same node become one passage
_h = [dict(content='first half', node_id='d#1', page=3, _rrf_score=0.02, rowid=1),
      dict(content='second half', node_id='d#1', page=4, _rrf_score=0.01, rowid=2),
      dict(content='elsewhere',   node_id='d#9', page=40, _rrf_score=0.005, rowid=3)]
_m = merge_spans(_h)
assert len(_m) == 2 and _m[0]['_nspan'] == 2 and 'second half' in _m[0]['content'], _m
assert _m[0]['_rrf_score'] == 0.02, 'a merged span keeps its best score, it does not accumulate'

# ingestion is content-addressed, and delete_doc leaves nothing behind
again = db.add_doc(DOC, title='Grahas', source='notes/grahas.md', emb_fn=enc)
assert 'skipped' in again, again
did = t['doc_id']
n_before = len(db.t.store())
db.delete_doc(did)
assert len(db.t.store()) == 0 and db.toc() == [] and n_before > 0
print(f'{n_before} chunks removed with the document')

5 chunks removed with the document

Limits

• No language model is used. A summary per node is what toc() shows a reader deciding where to look, and the default is only the first 300 characters. Pass summarize= to write better ones.
• Chunk size matters more than anything else in this module. In 07_doc_eval, cutting smaller chunks instead of page-sized ones raised MRR on reworded queries from 0.354 to 0.532, while breadcrumbs in the embedding and span joining changed it by less than 0.01. The default chunker is chunk_markdown; pass a better splitter as chunker=.

import tempfile
_d = Path(tempfile.mkdtemp())
(_d/'ratios.md').write_text('# Ratios\n\n## Liquidity\n\nCurrent ratio is current assets over current liabilities.')
(_d/'notes.txt').write_text('A plain text file with no headings at all, ingested as one windowed node.')
_fdb = database()
_fdb.get_tree('store')
print(_fdb.add_dir(_d, emb_fn=enc))
assert {d['title'] for d in _fdb.toc()} == {'ratios', 'notes'}
_rt = _fdb.toc('ratios')[0]['tree']
assert [n['title'] for n in _rt['children']] == ['Ratios']            # the `# Ratios` heading
assert [n['title'] for n in _rt['children'][0]['children']] == ['Liquidity']
assert _fdb.add_file(_d/'ratios.md', emb_fn=enc).get('skipped'), 're-ingest is a no-op'
assert 'current liabilities' in _fdb.read(_fdb.toc('ratios')[0]['tree']['id'])['text']

[{'doc_id': '1864a18be172a46b', 'title': 'notes', 'kind': 'txt', 'nodes': 2, 'chunks': 1}, {'doc_id': '20e07349293dcac7', 'title': 'ratios', 'kind': 'md', 'nodes': 3, 'chunks': 1}]