core

Building blocks for litesearch

Introduction

We often have to go through a whole bunch of hoops to get documents processed and ready for searching through them. litesearch plans to make this as easy as possible by providing simple building blocks to set up a database with FTS5 and vector search capabilities.

/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/site-packages/usearch/__init__.py:125: UserWarning: Will download `usearch_sqlite` binary from GitHub.
  warnings.warn("Will download `usearch_sqlite` binary from GitHub.", UserWarning)

---

source

Database.query

def query(
    sql:str, params:Union=None
)->Generator:

Execute a query and return results as a list of AttrDict

Simple Docs table setup

---

source

Database.get_store

def get_store(
    name:str='store', # table name
    hash:bool=False, # whether to create hash index on content
    kw:VAR_KEYWORD
):

Make a sql table for content storage with FTS5 and vector search capabilities

---

source

database

def database(
    pth_or_uri:str=':memory:', # the database name or URL
    wal:bool=True, # use WAL mode
    sem_search:bool=True, # enable usearch extensions
    kw:VAR_KEYWORD
)->Database: # additional args to pass to apswutils database

Set up a database connection and load usearch extensions.

---

source

Database.search

def search(
    q:str, # query string
    emb:bytes, # embedding vector
    columns:list=None, # columns to return
    where:str=None, # additional where clause
    where_args:dict=None, # args for where clause
    limit:int \| None=50, # limit on number of results
    offset:int \| None=None, # offset for results
    table_name:str='store', # table name
    emb_col:str='embedding', # embedding column name
    emb_metric:str='cosine', # embedding distance metric (cosine,sqeuclidean,inner,divergence)
    rrf:bool=True, # need to rerank results with reciprocal rank fusion
    dtype:type=<class 'numpy.float16'>, # embedding dtype
):

Search the litesearch store with fts and vector search combined.

Let’s test it out. We will create a database, run embedding comparisons, create a store and run search

db = database()

The fastlite database is set up with usearch extensions. Let’s run some distance calculations.

embs = dict(
    v1=np.ones((100,),dtype=np.float32).tobytes(),      # vector of ones
    v2=np.zeros((100,),dtype=np.float32).tobytes(),     # vector of zeros
    v3=np.full((100,),0.25,dtype=np.float32).tobytes()  # vector of 0.25s
)
def dist_q(metric):
    return db.q(f'''
        select
            distance_{metric}_f32(:v1,:v2) as {metric}_v1_v2,
            distance_{metric}_f32(:v1,:v3) as {metric}_v1_v3,
            distance_{metric}_f32(:v2,:v3) as {metric}_v2_v3
    ''', embs)

for fn in ['sqeuclidean', 'divergence', 'inner', 'cosine']: print(dist_q(fn))

[{'sqeuclidean_v1_v2': 100.0, 'sqeuclidean_v1_v3': 56.25, 'sqeuclidean_v2_v3': 6.25}]
[{'divergence_v1_v2': 34.657352447509766, 'divergence_v1_v3': 12.046551704406738, 'divergence_v2_v3': 8.66433334350586}]
[{'inner_v1_v2': 1.0, 'inner_v1_v3': -24.0, 'inner_v2_v3': 1.0}]
[{'cosine_v1_v2': 1.0, 'cosine_v1_v3': 0.0, 'cosine_v2_v3': 1.0}]

db.get_store()
if 'store' in db.t: print('store is created')
print('detected fts table: ',db.t.store.detect_fts())
print('Search results:', len(db.search('h',np.zeros((100,)).tobytes()))) # there is no data yet, so should be 0

store is created
detected fts table:  store_fts
Search results: 0

We can also create a store with hash index on content. Useful for code search applications

st=db.get_store(name='my_store', hash=True)
st.insert_all([dict(content='hello world', embedding=np.ones((100,),dtype=np.float16).tobytes()),
                           dict(content='hi there', embedding=np.full((100,),0.5,dtype=np.float16).tobytes()),
                           dict(content='goodbye now', embedding=np.zeros((100,),dtype=np.float16).tobytes())],upsert=True,hash_id='id')
st(select='id,content')

[{'id': '250ce2bffa97ab21fa9ab2922d19993454a0cf28', 'content': 'hello world'},
 {'id': 'c89f43361891bfab9290bcebf182fa5978f89700', 'content': 'hi there'},
 {'id': '882293d5e5c3d3e04e8e0c4f7c01efba904d0932', 'content': 'goodbye now'}]

Let’s run a search again.

db.search(q='hello', emb=np.full((100,),0.25, dtype=np.float16).tobytes(), columns=['content'], table_name='my_store',limit=2, rrf=False)

{'fts': [{'content': 'hello world'}],
 'vec': [{'content': 'hello world'}, {'content': 'hi there'}]}

Now, let’s try the same but with a broader query.

db.search(q='goodbye OR hi', emb=np.full((100,),0,dtype=np.float16).tobytes(), columns=['content'], table_name='my_store',limit=2)

[{'content': 'goodbye now'}, {'content': 'hello world'}]

You can use different kind of embedding metrics as well. The default is cosine. Let’s try with divergence distance

db.search(q='goodbye OR hi', emb=np.full((100,),0,dtype=np.float16).tobytes(), columns=['content'], table_name='my_store',limit=2, emb_metric='divergence')

[{'content': 'goodbye now'}, {'content': 'hi there'}]