# api ## What [`Index`](https://Karthik777.github.io/litesearch/api.html#index) decides Searching a folder used to take six decisions before the first result. [`Index`](https://Karthik777.github.io/litesearch/api.html#index) makes five of them from `evals/` and leaves one.
decision Index measured (weighted section MRR, 3 genres x 120 queries)
encoder static potion-multilingual-128M spread across four encoders is 0.018 to 0.046, and a static one wins astrology
dtype float16, matched to the store a mismatch returns rowid order with no error
granularity 512 characters +0.06 to +0.12 over page-sized
FTS leg pre(): keywords, wildcards, OR +0.016 to +0.093
vector leg HNSW ANN −0.005 quality, large speedup
tree always built ranking is a wash, −0.052 to +0.011, and toc/read/sections come free
rerank yours (rerank=True) +0.026 to +0.077, positive in all twelve paired cells
`Index.db` is the [`database()`](https://Karthik777.github.io/litesearch/core.html#database) you would have built by hand, and every method on it still works. ## [`Index`](https://Karthik777.github.io/litesearch/api.html#index) Three lines to a searchable corpus: ``` python ix = Index('kb.db') ix.add('docs/') ix.search('how does batching work') ``` [`Index`](https://Karthik777.github.io/litesearch/api.html#index) owns an encoder, a chunk store and a document tree. It is deliberately not clever: every method is a thin call into `litesearch.core` or `litesearch.tree` with the arguments the evaluation argues for already filled in. ------------------------------------------------------------------------ source ### Index ``` python def Index( path:str=':memory:', # sqlite file; omit for in-memory encoder:NoneType=None, # anything `doc_encoder` takes (default: static potion-multilingual-128M) name:str='store', # chunk table, if you want several corpora in one file ann:bool=True, # maintain an HNSW index for the vector leg db:NoneType=None, # an open Database to share, for several corpora in one file ): ``` *A corpus you can search: ingest files, code or text, then query with a string.* Every default is what `evals/` measures as best or tied-best across three genres. All of them are overridable, and `self.db` is the plain [`database()`](https://Karthik777.github.io/litesearch/core.html#database) underneath. ### Ingest `add` takes whatever you have: a directory, a single file, a string, a list of strings, or a `{title: text}` mapping. Documents go through `litesearch.tree`, so each keeps its heading structure and every chunk remembers the section it came from. Source code goes through `add_code` instead. Its tree is module › class › function and comes from the AST rather than from headings, a different parser, so a different door. ------------------------------------------------------------------------ source ### Index.add ``` python def add( src, # directory, file path, string, list of strings, or {title: text} types:str='.pdf,.md,.markdown,.txt,.rst,.ipynb,.xml,.tei,.htm,.html,.conllu', # extensions to pick up when `src` is a directory **kw )->int: # forwarded to add_dir / add_file / add_doc ``` *Ingest documents or text. Returns the number of chunks the store gained.* ------------------------------------------------------------------------ source ### Index.add_code ``` python def add_code( src, # a source directory, or an installed package name **kw )->int: # forwarded to dir2chunks / pkg2chunks ``` *Ingest source code through the AST path — top-level functions, classes and assignments.* ### Search `search` is the hybrid the whole library is for: FTS5 keyword and SIMD vector, merged with Reciprocal Rank Fusion. Pass a string, get chunks back. `rerank=True` is the one knob the evaluation asks you to think about. It fetches `RERANK_FANOUT` candidates and reorders them with a flashrank cross-encoder, worth +0.026 to +0.077 weighted MRR, positive in all twelve paired cells measured, at roughly 10x the latency and a small model download on first use. The fanout is load-bearing: reranking ten candidates only reorders ten, and the measured gain comes from reranking thirty. ------------------------------------------------------------------------ source ### Index.search ``` python def search( q:str, # query string limit:int=10, # hits to return rerank:bool=False, # reorder with a flashrank cross-encoder (see above) where:str=None, # SQL over the chunk store, to search part of it **kw )->list: # forwarded to Database.doc_search ``` *Hybrid keyword + vector search: `doc_search`, so hits are span-merged and carry a breadcrumb.* ### Read The tree is built at ingest whether or not you ask for it, because ranking-wise it is free. These four methods are what it buys, and the reason to keep it even though it does not move MRR.
method what it answers
toc() “what is even in this corpus?”, touches no embeddings
sections(q) “which sections are about this”, not which 512 characters
read(node_id) “give me that whole section”, reassembled from its chunks
context(q) operative sections plus what they connect to, composed for a model
On the Sanskrit corpus `context` roughly doubles verse-level recall over plain chunk search (0.190 → 0.340), the largest single effect measured anywhere in `evals/`, and larger than any encoder swap. Section assembly is doing real work even where section ranking is not. ------------------------------------------------------------------------ source ### Index.context ``` python def context( q:str, where:str=None, # SQL over the chunk store, applied to every retrieval leg **kw )->dict: ``` *Operative sections plus what they connect to, composed for a prompt.* ------------------------------------------------------------------------ source ### Index.read ``` python def read( node_id:str, store:str=None, # the store the node is in; None -> this one **kw )->dict: ``` *One whole section, reassembled — the unit to hand a model instead of fragments.* ------------------------------------------------------------------------ source ### Index.sections ``` python def sections( q:str, limit:int=5, # sections to return where:str=None, # SQL over the chunk store, to search part of it **kw )->list: ``` *Ranked sections rather than chunks, each with snippets and a `read` handle.* ------------------------------------------------------------------------ source ### Index.toc ``` python def toc( doc:str=None, **kw )->list: ``` *The corpus as a nested tree of titles and page ranges. Touches no embeddings.* ## Worked example Five short documents, ingested as raw text so the cell runs anywhere. ``` python ix = Index() ix.add({'Batching': 'Requests are batched by the scheduler before they reach the model. ' 'Batch size trades latency for throughput.', 'Caching': 'A prompt cache stores the prefix of a request so that repeated prefixes ' 'skip recomputation entirely.', 'Sharding': 'Large models are sharded across devices; each shard holds a slice of every ' 'weight matrix.', 'Metrics': 'Throughput is tokens per second. Latency is time to first token.', 'Retries': 'Failed requests are retried with exponential backoff and a jitter term.'}) ix ``` /Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm Index(path=':memory:', chunks=5, docs=5) ``` python [(h['doc_id'], h['content'][:44]) for h in ix.search('batch size and throughput', limit=3)] ``` [('63cecc4a6393c072', 'Requests are batched by the scheduler before'), ('341e7b88cc136169', 'Throughput is tokens per second. Latency is '), ('3cc7b6b520ad1a39', 'A prompt cache stores the prefix of a reques')] The same query rolled up to sections rather than chunks: ``` python [(s['node_id'], (s['snippets'] or [''])[0][:44]) for s in ix.sections('batch size and throughput', limit=2)] ``` [('63cecc4a6393c072#1', 'Requests are batched by the scheduler before'), ('341e7b88cc136169#1', 'Throughput is tokens per second. Latency is ')] ``` python [d['title'] for d in ix.docs] ``` ['Batching', 'Caching', 'Sharding', 'Metrics', 'Retries'] ### Several corpora in one file `db=` hands an [`Index`](https://Karthik777.github.io/litesearch/api.html#index) a `Database` that is already open, so two [`Index`](https://Karthik777.github.io/litesearch/api.html#index)es can be two stores in one file. They must share the connection, not just the path: `database(':memory:')` called twice is two different databases, and even on disk two connections is two write locks. Each store keeps its own chunks, its own tree and its own ANN index, so a query against one never sees the other. What they cannot do is share a vector space, an ANN index holds one encoder’s output, and two encoders’ vectors compared as bytes give distances that mean nothing. Partitioned this way, each ranks correctly on its own and you combine them by rank. ``` python dbf = database() # one file, opened once main, papers = Index(db=dbf), Index(db=dbf, name='papers') main.add({'Batching': 'Throughput rises with batch size until the device runs out of memory.'}) papers.add({'Scaling': 'Loss follows a power law in parameters, data and compute.'}) ([d['title'] for d in main.docs], [d['title'] for d in papers.docs]) ``` (['Batching'], ['Scaling']) ## The layer underneath [`Index`](https://Karthik777.github.io/litesearch/api.html#index) is one of two routes and the other is not going away. Reach past it when you want something it deliberately decided for you:
you want use
SQL over your own columns, joins, filters ix.db, a plain database()
a different encoder, or float32 vectors database() + get_store() directly
several stores, custom FTS tokenizers db.get_store(name=..., tokenize=...)
entity-graph retrieval for bridge queries db.graph_search, from vruksha
Sanskrit script folding litesearch.sanskrit, see sanskrit
Sanskrit metre, verse trees, lemmas ganapati
The graph leg is the one capability [`Index`](https://Karthik777.github.io/litesearch/api.html#index) does not expose at all, and that is measured rather than an oversight. On ordinary known-item queries it costs 0.070 to 0.160 weighted MRR, negative in every cell, every genre and every flavour, monotonically worse as its weight rises. On the bridge query set built specifically to favour it, where the answer shares no token with the question, it buys roughly +0.04 target MRR and +0.12 hit@1, on one genre of three, while losing 0.10 to 0.16 on the ordinary questions and running 3–4x slower. Call `db.graph_search` by name when you know your traffic looks like that. Do not reach for it by default. ## Downstream: vishalakshi is the first test [vishalakshi](https://github.com/vedicreader/vishalakshi) builds a litesearch-backed vault, and it is the nominated candidate for porting onto [`Index`](https://Karthik777.github.io/litesearch/api.html#index), a real caller is a better test of “did this actually remove decisions” than any example in this notebook. What the port should tell us: - **Does `add` cover the ingest it does by hand?** The Sanskrit [`Profile`](https://Karthik777.github.io/litesearch/data.html#profile)s register at import, so `add_file` already picks the reader, `verse` tree mode, `VerseChunker` and the metrical facets without arguments. [`Index.add`](https://Karthik777.github.io/litesearch/api.html#index.add) forwards to it unchanged, which means the vault should need no chunker and no tree wiring of its own. If it does, that is a gap in [`Index`](https://Karthik777.github.io/litesearch/api.html#index), not in the caller. - **Is the default encoder wrong for it?** [`Index`](https://Karthik777.github.io/litesearch/api.html#index) defaults to `potion-multilingual-128M`, to enable vishalakshi on the Gītā full-stack table - **Does `context()` carry the vault’s read path?** It is the method with the largest measured effect on that corpus (0.190 → 0.340 verse recall against plain chunk search), so a vault that assembles passages for a reader should be built on `context`/`read`, not on `search`. Note this was written without reading vishalakshi’s source, so treat the three points as the questions to answer during the port rather than as findings about its current code. ``` python th#| hide import nbdev; nbdev.nbdev_export() ```