# store ## Stored arrays Some search structures are expensive to build and cheap to read: a matrix of every embedding in a store, or a lookup table of folded spellings. A web app usually runs several worker processes, and if each one builds the structure itself, it pays the build time at every start and holds its own copy in memory. [`stored`](https://Karthik777.github.io/litesearch/store.html#stored) builds the arrays once, saves them as `.npy` files under `root/-/`, and returns them memory-mapped. A memory-mapped array is read from the file by the operating system as pages are touched, instead of being loaded into the process. Every process that maps the same file shares those pages, so ten workers reading a 100 MB matrix use about 100 MB between them, not 1 GB. `key` says what the arrays were built from. Choose something that changes whenever the source data changes: an index’s FTS signature, a row count with the highest rowid, plus the versions of the libraries that shaped the build. When the key changes, [`stored`](https://Karthik777.github.io/litesearch/store.html#stored) builds a new set and removes the old sets of the same `name`. A process that still has an old set mapped keeps reading it safely, since a deleted file stays readable while it is mapped. The first process to need a set takes a file lock, builds the arrays into a temporary folder and renames the folder into place. Other processes starting at the same moment wait on the lock and then map the finished set, so they never see half-written files and the build runs once. If `root` cannot be written, [`stored`](https://Karthik777.github.io/litesearch/store.html#stored) builds the arrays in memory and returns them unshared. Call it without `build` in a process that should only read: it returns the set if one exists for that key, and None if not. ------------------------------------------------------------------------ source ### stored ``` python def stored( root, # folder that holds every stored set name, # what the arrays are; the prefix of their folder key, # what the arrays are built from; a new key triggers a rebuild build:NoneType=None, # `() -> {name: array}`; None only maps an existing set, or returns None )->dict: ``` *Return `build()`’s arrays memory-mapped from `root/-`, building them once across all processes.* ``` python _root = Path(tempfile.mkdtemp()) _n = [0] def _b(): _n[0] += 1; return dict(a=np.arange(5), b=np.array(['x', 'y'])) _s = stored(_root, 'demo', 'k1', _b) test_eq(_s['a'].tolist(), [0, 1, 2, 3, 4]); assert isinstance(_s['a'], np.memmap) stored(_root, 'demo', 'k1', _b); test_eq(_n[0], 1) # built once stored(_root, 'demo', 'k2', _b); test_eq(_n[0], 2) test_eq(sorted(p.name for p in _root.glob('demo-*')), ['demo-k2']) # the older set is gone test_eq(stored(_root, 'demo', 'k1'), None) # no build: only what is there test_eq(stored(_root, 'demo', 'k2')['a'].tolist(), [0, 1, 2, 3, 4]); test_eq(_n[0], 2) ``` ``` python _pr = Path(tempfile.mkdtemp()) _src = f"""import numpy as np, time; from pathlib import Path; from litesearch.store import stored def b(): open(Path({str(_pr)!r})/'built', 'a').write('x'); time.sleep(.3); return dict(a=np.ones(3)) stored({str(_pr)!r}, 'par', 'k', b)""" _ps = [subprocess.Popen([sys.executable, '-c', _src]) for _ in range(3)] test_eq([p.wait() for p in _ps], [0, 0, 0]) test_eq((_pr/'built').read_text(), 'x') # three processes, one build ``` ## A key index FTS5 matches the tokens its tokenizer produced. Sometimes an app needs to find rows by a key the tokenizer never made: an alias for a name, a romanised title, or a spelling folded by a rule of your own. [`KeyIndex`](https://Karthik777.github.io/litesearch/store.html#keyindex) is an inverted index for that case. For each key it keeps a posting list, the list of row numbers that hold the key, so a lookup returns rows directly without scanning text. `KeyIndex.build(pairs)` takes `(key, row)` pairs and returns plain arrays, which you can pass to [`stored`](https://Karthik777.github.io/litesearch/store.html#stored) so every process shares them. `KeyIndex(arrays)` wraps the arrays for reading. The keys are sorted and packed into one UTF-8 byte string, so `lookup(key)` is a binary search. It returns the rows holding `key`, or with `prefix=True` the rows of every key that starts with `key`, sorted and without repeats. Use FTS5 for words in running text. Use [`KeyIndex`](https://Karthik777.github.io/litesearch/store.html#keyindex) when you already know the exact keys each row should answer to. ------------------------------------------------------------------------ source ### KeyIndex ``` python def KeyIndex( a ): ``` *Look up row numbers by string key, exactly or by prefix, from arrays made by [`KeyIndex.build`](https://Karthik777.github.io/litesearch/store.html#keyindex.build).* ``` python _k = KeyIndex(KeyIndex.build([('krsna', 0), ('krsna', 2), ('rama', 1), ('ramayana', 3), ('कृष्ण', 2)])) test_eq(_k.lookup('krsna').tolist(), [0, 2]) test_eq(_k.lookup('rama').tolist(), [1]) test_eq(_k.lookup('rama', prefix=True).tolist(), [1, 3]) test_eq(_k.lookup('कृ', prefix=True).tolist(), [2]) test_eq(_k.lookup('sita').tolist(), []) test_eq(len(KeyIndex(KeyIndex.build([]))), 0) _ks = KeyIndex(stored(_root, 'keys', 'k', lambda: KeyIndex.build([('a', 1), ('ab', 2)]))) test_eq(_ks.lookup('a', prefix=True).tolist(), [1, 2]) ``` ## A vector matrix There are two ways to find the stored vectors nearest a query vector. Exact search compares the query with every vector and returns the true nearest. Approximate nearest-neighbour (ANN) search walks a prebuilt graph, here usearch’s HNSW index from `get_store(ann=True)`, and returns nearly the nearest in far fewer comparisons. ANN is the right choice for a very large store; it costs an index that has to be built and kept in step with the table. [`VecMatrix`](https://Karthik777.github.io/litesearch/store.html#vecmatrix) is exact search done fast. It holds a store’s embeddings as one float32 matrix with every row scaled to unit length, so one matrix product with the normalised query gives the cosine similarity of every row at once. There is no index to keep in step. `vec_search` in `litesearch.core` is also exact, but it computes each distance in SQL on every query; [`VecMatrix`](https://Karthik777.github.io/litesearch/store.html#vecmatrix) reads the vectors once and keeps them in memory, or memory-mapped through [`stored`](https://Karthik777.github.io/litesearch/store.html#stored). The matrix takes 4 bytes per dimension per row, so 100,000 rows of 256 dimensions is about 100 MB. `VecMatrix.build(tbl)` reads the vectors in rowid order, optionally narrowed by a SQL `where`, and returns the arrays. Pass the `dtype` the vectors were stored as; the default is float16. `search(q, k)` returns `(rowid, similarity)` pairs, most similar first, with equal scores in row order. `rows=` limits the search to some positions in the matrix, such as the rows of one book. These are positions, not rowids. Embeddings have no FTS signature, so pick the [`stored`](https://Karthik777.github.io/litesearch/store.html#stored) key yourself: the row count and the highest rowid, or a counter your own writes increase. ------------------------------------------------------------------------ source ### VecMatrix ``` python def VecMatrix( a ): ``` *Search a store’s embeddings by exact cosine similarity, held as one unit-length float32 matrix.* ``` python _vd = database(sem_search=False) _vd.execute('create table v(name text, embedding blob)') for nm, e in [('a', [1, 0, 0]), ('b', [0, 1, 0]), ('c', [1, 1, 0]), ('none', None)]: _vd.execute('insert into v values(?, ?)', [nm, None if e is None else np.array(e, np.float16).tobytes()]) _vm = VecMatrix(VecMatrix.build(_vd.t.v)) test_eq(len(_vm), 3) _hits = _vm.search(np.array([1, .1, 0], np.float16).tobytes(), k=2) test_eq([i for i, _ in _hits], [1, 3]); assert _hits[0][1] > _hits[1][1] test_eq(VecMatrix(VecMatrix.build(_vd.t.v, where="name='b'")).search([0, 1, 0], k=5)[0][0], 2) _vs = VecMatrix(stored(_root, 'vecs', 'k', lambda: VecMatrix.build(_vd.t.v))) assert isinstance(_vs.m, np.memmap) and _vs.search([0, 1, 0], k=1)[0][0] == 2 test_eq(VecMatrix(VecMatrix.build(_vd.t.v, where='0')).search([1, 0, 0]), []) test_eq([i for i, _ in _vm.search([1, 0, 0], k=5, rows=[1, 2])], [3, 2]) # within matrix rows 1 and 2: c, then b test_eq(_vm.search([1, 0, 0], rows=[]), []) _vd.execute('create table vi(id integer primary key, embedding blob)') # rowid read under its alias's name _vd.execute('insert into vi values(7, ?)', [np.array([1, 0, 0], np.float16).tobytes()]) test_eq(VecMatrix(VecMatrix.build(_vd.t.vi)).search([1, 0, 0])[0][0], 7) _vt = VecMatrix(dict(ids=np.arange(4), m=np.array([[1, 0], [0, 1], [0, 1], [0, 1]], np.float32))) test_eq([i for i, _ in _vt.search([0, 1], k=2)], [1, 2]) # equals in row order ```