litesearch

search through files with fts5, vectors and get reranked results. Fast.

This file will become your README and also the index of your documentation.

Developer Guide

If you are new to using nbdev here are some useful pointers to get you started.

Install litesearch in Development mode

# make sure litesearch package is installed in development mode
$ pip install -e .

# make changes under nbs/ directory
# ...

# compile to have changes apply to litesearch
$ nbdev_prepare

Usage

Installation

Install latest from the GitHub repository:

$ pip install git+https://github.com/Karthik777/litesearch.git

or from pypi

$ pip install litesearch

Documentation

Documentation can be found hosted on this GitHub repository’s pages. Additionally you can find package manager specific guidelines on conda and pypi respectively.

Let’s setup some deps to make full use of litesearch

from fastcore.all import *
from fastlite import *
import numpy as np
from litesearch import *

Let’s set the db up. This db has usearch loaded. So, you can run cosine distance calculations using simd(means fast, real fast)

db: Database = setup_db(':memory:')
embs = dict(v1=np.ones(512).tobytes(), v2=np.zeros(512).tobytes())
db.q('''select
    distance_cosine_f16(:v1,:v2) as diff,
    distance_cosine_f16(:v1,:v1) as same ''',embs)

[{'diff': 1.0, 'same': 0.0}]

There are way more functions you can run now. Checkout: https://unum-cloud.github.io/USearch/sqlite/index.html

Checkout the examples/01_simple_rag.ipynb for a full-fledged rag example.

Let’s create a store and push some content in.

store = db.mk_store()

store.schema

'CREATE TABLE [content] (\n   [id] INTEGER PRIMARY KEY,\n   [content] TEXT NOT NULL,\n   [embedding] BLOB,\n   [metadata] TEXT,\n   [uploaded_at] FLOAT DEFAULT CURRENT_TIMESTAMP\n)'

txts = ['this is a text', "I'm hungry", "Let's play! shall we?"]
embs = [np.full(512, i) for i in range(3)]
rows = [dict(content=t, embedding=e) for t,e in zip(txts,embs)]
store.insert_all(rows)

<Table content (id, content, embedding, metadata, uploaded_at)>

Cool, let’s search through these contents

litesearch provides a search method which reranks the results from both FTS and vector search using Reciprocal Rank Fusion (RRF)

You can always turn it off.

q,e='playing hungry',np.full(512,1).tobytes()
res = db.search(pre(q), e, columns=['id', 'content'], lim=2)
print(res)

[{'id': 2, 'content': "I'm hungry"}, {'id': 3, 'content': "Let's play! shall we?"}, {'id': 1, 'content': 'this is a text'}]