This module groups the chunks in a store by meaning and gives each group a short name. It works from the vectors already in the store’s ANN index, so it needs no extra model.
An ANN (approximate nearest neighbour) index is the usearch HNSW index that get_store(ann=True) attaches to a store. It finds the vectors closest to a query vector without comparing the query against every row. The functions here reuse the same index to find which chunks sit close to each other.
get_graph creates three tables beside a chunk store, for recording named things (entities) and where they occur:
| table | one row per | columns |
|---|---|---|
entities |
named thing, such as a topic | content (the name), kind, canon, freq |
mentions |
chunk that mentions an entity | chunk_id, entity_id, surface, n |
edges |
relation between two entities | src, dst, rel, weight, n |
entities is an ordinary store made by get_store, so it has full-text search and, with ann=True, its own ANN index. For the default store the tables are called entities, mentions and edges; for a store called docs they are docs_entities and so on. Calling get_graph again on the same store changes nothing and returns the same tables.
def get_graph(
store:str='store', # chunk store the graph is built over
prefix:str=None, # table prefix (default: '' for 'store', else '<store>_')
ann:bool=True, # register an ANN index on the entity store (needed for resolution)
encoder:str=None, # the encoder whose vectors the entity store holds; checked on reopen
**kw
):Create the entities, mentions and edges tables for a chunk store, or open them if they exist.
A cluster is a group of chunks whose vectors lie close together, which in practice means chunks about the same subject. Two methods find clusters:
Index too small to cluster!.k nearest neighbours (8 by default), then builds groups greedily: it starts from the vector whose close neighbours have the highest total similarity, adds its unassigned neighbours to the group, and repeats with the next unassigned vector. A neighbour counts as close when its cosine similarity is at least the median over all neighbour pairs. Each group holds at most max(k, 4) members, so one large subject comes out as several groups of that size. This method works on a store of any size from 4 vectors up.Every result reports which method ran, as method='usearch' or method='knn'.
Each cluster is named by c-TF-IDF (class-based term frequency, inverse document frequency). A word scores high for a cluster when many of the cluster’s members use it and few other clusters do. Naming by plain word counts would give every cluster the same few common words. A word that appears in every cluster is never used in a name, and words under three letters and common English words (_STOP) are skipped.
ctfidf_labels does the naming and works on any texts, not only store rows: pass one text per member and the cluster number of each member. topic_nodes clusters a store and writes each cluster into the graph tables as a topic entity, with a mention row linking it to each member chunk.
def topic_nodes(
db, # Database
store:str='store', # chunk store (must be ANN-registered)
prefix:NoneType=None, # graph table prefix
min_count:NoneType=None, # usearch cluster min size
max_count:NoneType=None, # usearch cluster max size
k:int=8, # neighbours per node in the fallback kNN graph
min_size:int=2, # smallest cluster kept as a topic
label_k:int=4, # words per topic name
max_label_chars:int=4000, # characters of each member's text used for naming
dtype:type=float16
):Cluster a store and save each cluster as a named topic entity in its graph tables, returning {topics, method}.
Name each cluster by the words common among its members and rare in the other clusters.
These two methods on a store answer questions about the shape of the corpus rather than about a query.
store.clusters() divides the whole store into named groups. Use it to show a reader what a corpus is about, or to browse it by subject. It returns AttrDict(clusters, method, note). Each cluster is AttrDict(centroid, size, label, member_keys, members): member_keys holds the rowids of every member, members holds the rows of the first 24 (set by members=), and the list is sorted largest cluster first.
store.peers(rowid) returns the other members of the cluster that one row belongs to. It returns AttrDict(hits, method, note), with the members in cluster order.
peers and ann_neighbors both start from a row, and they answer different questions. ann_neighbors returns a fixed number of nearest rows (15 by default), however far away the last one is. peers returns the row’s group, which may be smaller and only holds rows that clustered together. When peers cannot use a cluster, because the store has fewer than 4 vectors, the row was left in no cluster, or its cluster has only itself, it returns the limit nearest rows from ann_neighbors instead and sets method='ann'.
Both methods always fill note with a sentence saying what happened, such as 6 clusters over 60 vectors (knn) or index holds 1 vectors; too few to cluster. An empty result looks the same whether the store is too small, has no vectors yet, or has no ANN index, and note tells them apart.
The clustering is computed once and cached on the connection. It is recomputed when the index changes size or when you pass different min_count, max_count or k, so calling peers for many rows in a row is cheap.
The example below builds two clearly separated sets of 30 vectors, one for texts about tensors and one about invoices. The fallback method splits each set into groups of up to 8, and every group is named after the words of its own set.
def peers(
key:int, # usearch key (rowid) whose group you want
limit:int=25, # members to return
columns:list=None, # store columns to return per member row
min_count:int=None, # usearch: smallest cluster to emit
max_count:int=None, # usearch: largest cluster to emit
k:int=8, # neighbours per node in the kNN fallback
dtype:type=float16
):Return the other members of the cluster that row key belongs to.
def clusters(
min_count:int=None, # usearch: smallest cluster to emit
max_count:int=None, # usearch: largest cluster to emit
k:int=8, # neighbours per node in the kNN fallback
min_size:int=2, # drop groups smaller than this
label_k:int=4, # words per cluster name
members:int=24, # member rows fetched per cluster
columns:list=None, # store columns to return per member row
max_label_chars:int=4000, # characters of each member's text used for naming
dtype:type=float16
):Divide the store into clusters of similar chunks, each named by the words that set it apart.
# clusters(): two well-separated blobs -> two groups, each labelled by its own vocabulary
from litesearch.core import database
_cdb = database()
_cst = _cdb.get_store('cl', ann=True, ndim=8, metric='cosine')
_rs = np.random.RandomState(0)
_a = np.concatenate([np.ones((30,4)), np.zeros((30,4))], 1) + _rs.randn(30,8)*0.05
_b = np.concatenate([np.zeros((30,4)), np.ones((30,4))], 1) + _rs.randn(30,8)*0.05
_txt = [f'kernel gradient tensor sample {i}' for i in range(30)] + [f'invoice ledger payment row {i}' for i in range(30)]
_cst.insert_all([{'content':t,'embedding':v.astype(np.float16).tobytes()}
for t,v in zip(_txt, np.concatenate([_a,_b]))])
assert _cst.rebuild_index() == 60
_cl = _cst.clusters(min_size=3)
print(_cl.note, '|', [(c.size, c.label) for c in _cl.clusters][:4])
assert len(_cl.clusters) >= 2, _cl.note
assert _cl.method in ('usearch','knn')
_labels = ' '.join(c.label for c in _cl.clusters)
assert 'tensor' in _labels or 'kernel' in _labels, _labels # c-TF-IDF names a group after what only it says
assert all(r['rowid'] in c.member_keys for c in _cl.clusters for r in c.members)
# peers(): every member of a group has the rest of that group as its family
_c0 = first(_cl.clusters, lambda c: 'kernel' in c.label or 'tensor' in c.label)
_pr = _cst.peers(_c0.centroid, limit=5, columns=['content'])
print(_pr.note, '|', [h['content'][:24] for h in _pr.hits])
assert _pr.hits and _pr.method == _cl.method, _pr.note
assert all('kernel' in h['content'] for h in _pr.hits), _pr.hits
# a row the clustering left on its own still gets an answer -- with the fallback named in `note`
_solo = first(range(1, 61), lambda r: not _cst.peers(r).method == _cl.method)
if _solo: assert 'nearest neighbours' in _cst.peers(_solo).note
# degradation is reported, never silent
_tiny = database().get_store('tiny', ann=True, ndim=4, metric='cosine')
_tiny.insert_all([{'content':'x','embedding':np.zeros(4,dtype=np.float16).tobytes()}])
_tiny.rebuild_index()
assert _tiny.clusters().clusters == [] and 'too few' in _tiny.clusters().note
assert 'nearest neighbours' in _tiny.peers(1).note
assert database().get_store('plain').clusters().note.endswith('is not an ANN store')
assert database().get_store('empty', ann=True).clusters().note.endswith('has no vectors yet')6 clusters over 60 vectors (knn) | [(8, 'invoice, payment, ledger, row'), (8, 'gradient, tensor, sample, kernel'), (7, 'gradient, tensor, sample, kernel'), (5, 'invoice, payment, ledger, row')]
cluster of 8 (knn) | ['kernel gradient tensor s', 'kernel gradient tensor s', 'kernel gradient tensor s', 'kernel gradient tensor s', 'kernel gradient tensor s']/opt/hostedtoolcache/Python/3.12.15/x64/lib/python3.12/site-packages/usearch/__init__.py:131: UserWarning: Will download `usearch_sqlite` binary from GitHub.
warnings.warn("Will download `usearch_sqlite` binary from GitHub.", UserWarning)