api

Add documents, search them and read whole sections, with defaults chosen by measurement

What Index decides for you

Building search on the lower-level database() means choosing an encoder, a vector type, a chunk size, how to query, and whether to build a document tree, before the first result. Index makes all of these choices from the measurements in evals/ and leaves you one: whether to rerank.

The scores below are weighted section MRR over three corpora with 120 queries each. MRR (mean reciprocal rank) is 1 when the right section comes first, 1/2 when it comes second, and so on, averaged over the queries.

decision what Index uses what was measured
encoder the static model potion-multilingual-128M four encoders differ by only 0.018 to 0.046, and a static one is best on the astrology corpus
vector type float16, the type the store holds the wrong type returns rows in table order, with only a warning
chunk size 512 bytes (512 characters of English, about 170 of Devanagari, at three bytes a letter) +0.06 to +0.12 over page-sized chunks
keyword query pre(): drop stopwords, match word prefixes, join with OR +0.016 to +0.093
vector search an HNSW index -0.005, for a large speedup
document tree always built no effect on ranking (-0.052 to +0.011), and it makes toc, read and sections possible
rerank your choice, rerank=True +0.026 to +0.077, better in all twelve paired cells

Index.db is the database() you would have built by hand, and every function from the lower level works on it.

Index

Three lines give you a searchable collection:

ix = Index('kb.db')
ix.add('docs/')
ix.search('how does batching work')

An Index holds an encoder, a store of chunks and a document tree. Each method is a short call into litesearch.core or litesearch.tree with the measured defaults already filled in, so anything Index does you can also do by hand with those modules.

encoder can be any object with an encode method that takes a list of strings, or a FastEncode. The store records the encoder’s name when it has one (see encoder_id), and reopening the file with a differently named encoder raises StoreMismatch, because vectors from two models cannot be compared.

source

Index

def Index(
    path:str=':memory:', # sqlite file; omit for in-memory
    encoder:NoneType=None, # an object with `.encode`, or a `FastEncode` (default: static potion-multilingual-128M)
    name:str='store', # chunk table, if you want several corpora in one file
    ann:bool=True, # maintain an HNSW index for the vector leg
    db:NoneType=None, # an open Database to share, for several corpora in one file
):

A searchable collection: add files, code or text, then query it with a string.

Adding documents

add takes what you have: a directory, a file path, a string, a list of strings, or a {title: text} dict. Documents go through litesearch.tree, which keeps each document’s heading structure, so every chunk knows the section it came from. For a directory, types lists the file extensions to pick up. add returns the change in the store’s chunk count: 0 when every text was already stored, and negative when an edit shortened a document. One text is stored once. A file’s source is its resolved path, a copy or a string with stored text only records its source, and a file edited since it was added replaces its old chunks. Each hit carries its document’s source, and docs lists every source a document is stored for. remove(src) forgets a file, URL or title; its text stays while another source shares it. The tree page has the full contract.

Source code goes through add_code instead. It reads a directory or an installed package. Python files are cut at top-level functions and classes using Python’s own parser, other source files become one chunk per function signature (through codesigs), and any notebooks, Markdown, text or PDF files it finds are chunked as text. Each file becomes one document of kind code with a single node, under the same rules as add: a file re-added unchanged or copied writes nothing, and an edited file replaces its chunks. Each chunk’s file path and definition name are kept in its metadata.

source

Index.add

def add(
    src, # directory, file path, string, list of strings, or {title: text}
    types:str='.pdf,.md,.markdown,.txt,.rst,.ipynb,.xml,.tei,.htm,.html,.conllu', # extensions to pick up when `src` is a directory
    **kw
)->int: # forwarded to add_dir / add_file / add_doc

Add documents or text to the index and return the change in its chunk count, negative when an edit shortened a document.

source

Index.add_code

def add_code(
    src, # a source directory, or an installed package name
    force:bool=False, # rewrite files whose code is already stored
    **kw
)->int: # forwarded to dir2chunks / pkg2chunks

Add the source code in a directory or installed package, one document per file cut at top-level definitions, and return the number of chunks written.

Searching

search runs the hybrid search: an FTS5 keyword query and a vector search, merged with Reciprocal Rank Fusion. Pass a string and get chunks back, best first. Neighbouring hits from the same section are merged into one, and each hit has a breadcrumb, the path of headings above it, so you can cite where it came from. where takes a SQL condition on the store’s columns, such as doc_id='...', to search part of the collection.

rerank=True is the one setting the evaluation leaves to you. It takes RERANK_FANOUT (30) candidates and reorders them with a flashrank cross-encoder, a model that reads the query and each candidate together. That gained 0.026 to 0.077 weighted MRR, better in all twelve paired cells measured, at roughly 10 times the query time and a small model download the first time. The number of candidates matters: reranking ten candidates can only reorder those ten, and the measured gain comes from reranking thirty.

source

Index.search

def search(
    q:str, # query string
    limit:int=10, # hits to return
    rerank:bool=False, # reorder the top 30 candidates with a flashrank cross-encoder
    where:str=None, # SQL over the chunk store, to search part of it
    **kw
)->list: # forwarded to Database.doc_search

Hybrid keyword and vector search, returning chunks with neighbouring hits merged and a breadcrumb on each.

Reading

The document tree is built at ingest whether you use it or not, because it costs nothing in ranking. It is what these four methods need, and they are the reason to keep it.

method what it answers
toc() what is in the collection, as a nested table of contents. It reads no embeddings
sections(q) which sections are about the query, rather than which 512-byte chunks
read(node_id) the whole text of one section, put back together from its chunks
context(q) the sections that best answer the query, with their text and position in the tree, plus related sections, ready to give a language model

On the Sanskrit corpus, context nearly doubles verse-level recall over plain chunk search, from 0.190 to 0.340. That is the largest single effect measured anywhere in evals/, larger than any change of encoder. Putting sections back together helps even though ranking by section does not.

source

Index.context

def context(
    q:str, where:str=None, # SQL over the chunk store, applied to every retrieval leg
    **kw
)->dict:

The sections that best answer q with their text and tree position, plus related sections, for a prompt.

source

Index.read

def read(
    node_id:str, store:str=None, # the store the node is in; None -> this one
    **kw
)->dict:

The full text of section node_id, with its title, breadcrumb, pages and child sections.

source

Index.sections

def sections(
    q:str, limit:int=5, # sections to return
    where:str=None, # SQL over the chunk store, to search part of it
    **kw
)->list:

The sections that best match q, each with snippets and the node_id to pass to read.

source

Index.toc

def toc(
    doc:str=None, **kw
)->list:

The collection as nested tables of contents, one per document, with titles and page ranges.

Worked example

Five short documents, added as text so the cell runs anywhere.

ix = Index()
ix.add({'Batching': 'Requests are batched by the scheduler before they reach the model. '
                    'Batch size trades latency for throughput.',
        'Caching':  'A prompt cache stores the prefix of a request so that repeated prefixes '
                    'skip recomputation entirely.',
        'Sharding': 'Large models are sharded across devices; each shard holds a slice of every '
                    'weight matrix.',
        'Metrics':  'Throughput is tokens per second. Latency is time to first token.',
        'Retries':  'Failed requests are retried with exponential backoff and a jitter term.'})
ix
/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
Index(path=':memory:', chunks=5, docs=5)
[(h['doc_id'], h['content'][:44]) for h in ix.search('batch size and throughput', limit=3)]
[('63cecc4a6393c072', 'Requests are batched by the scheduler before'),
 ('341e7b88cc136169', 'Throughput is tokens per second. Latency is '),
 ('3cc7b6b520ad1a39', 'A prompt cache stores the prefix of a reques')]

The same query, ranked by section instead of by chunk:

[(s['node_id'], (s['snippets'] or [''])[0][:44])
 for s in ix.sections('batch size and throughput', limit=2)]
[('63cecc4a6393c072#1', 'Requests are batched by the scheduler before'),
 ('341e7b88cc136169#1', 'Throughput is tokens per second. Latency is ')]
[d['title'] for d in ix.docs]
['Batching', 'Caching', 'Sharding', 'Metrics', 'Retries']

Several collections in one file

db= gives an Index a database that is already open, so two Index objects can be two stores in one file. They must share the connection, not only the path: database(':memory:') called twice gives two separate databases, and two connections to one file on disk compete for one write lock.

Each store keeps its own chunks, its own tree and its own HNSW index, so a query on one never sees the other. They cannot share a vector space, though. An HNSW index holds one encoder’s vectors, and distances between two encoders’ vectors mean nothing. If the two collections use different encoders, search each one separately and merge the results by rank, for example with rrf_all.

dbf = database()                                    # one file, opened once
main, papers = Index(db=dbf), Index(db=dbf, name='papers')
main.add({'Batching': 'Throughput rises with batch size until the device runs out of memory.'})
papers.add({'Scaling': 'Loss follows a power law in parameters, data and compute.'})
([d['title'] for d in main.docs], [d['title'] for d in papers.docs])
(['Batching'], ['Scaling'])

Going below Index

Index is one of two routes, and the other stays available. Go below it when you need something it decides for you:

you want use
SQL over your own columns, joins, filters ix.db, a plain database()
a different encoder, or float32 vectors database() and get_store() directly
several stores, or a custom FTS5 tokenizer db.get_store(name=..., tokenize=...)
entity-graph retrieval for questions that share no words with the answer db.graph_search, added by vruksha when you import it
matching Sanskrit across scripts litesearch.sanskrit, see sanskrit
Sanskrit metre, verse trees, lemmas ganapati

Index does not offer the graph search at all, because it measured negative. On ordinary known-item queries it lowers weighted MRR by 0.070 to 0.160. On a set of 120 questions per corpus built to favour it, where the answer shares no word with the question, it raised target MRR by only 0.004 and 0.014 on the two corpora tried, while the score for the easy part of each question, the passage keyword search finds directly, fell by about 21% and taking 3 to 5 times as long. evals/RESULTS.md has the method. Use db.graph_search only when you know your questions look like that.

vishalakshi

vishalakshi is a document vault built on litesearch. Its Vault class is a subclass of Index, and its search, sections, context and read call Index’s methods. It is the largest user of this API, so a change to Index should be checked against it.