Three tables beside the chunk store. Entities live in a regular get_store so they inherit the existing ANN machinery, resolution reuses ann_search/rebuild_index rather than adding a second index path.
Create the entity/mention/edge tables for a chunk store. Idempotent; returns the tables.
usearch clusters off the HNSW graph but walks index levels, so it raises Index too small to cluster! on a small corpus. _knn_clusters falls back to a greedy pass over the kNN graph, which works at any size.
Labels are c-TF-IDF: term frequency inside a cluster, weighted by inverse document frequency across clusters. Plain frequency names every cluster after the same few words.
_knn_clusters seeds at the densest unassigned node and claims its unassigned neighbours, so clusters stay bounded. Label propagation collapses a dense kNN graph into one component.
def topic_nodes(
db, # Database
store:str='store', # chunk store (must be ANN-registered)
prefix:NoneType=None, # graph table prefix
min_count:NoneType=None, # usearch cluster min size
max_count:NoneType=None, # usearch cluster max size
k:int=8, # neighbours per node in the fallback kNN graph
min_size:int=2, # smallest cluster kept as a topic
label_k:int=4, # terms per topic label
max_label_chars:int=4000, dtype:type=float16
):Cluster the store index into topic nodes labelled by c-TF-IDF. Returns {topics, method}.
Name each cluster by terms common inside it and rare across the other clusters.
store.clusters() maps the corpus into labelled groups. store.peers(rowid) returns the group one row belongs to.
peers is not ann_neighbors. k-NN returns 15 nearest things whether or not they are related; a cluster returns the family. peers falls back to ann_neighbors when the index cannot be clustered, and says so in note.
Both return note, because an empty clustering and a broken index look identical to a caller.
clusters returns AttrDict(clusters, method, note), each cluster AttrDict(centroid, size, label, member_keys, members). method is usearch or knn.
peers returns AttrDict(hits, method, note).
def peers(
key:int, # usearch key (rowid) whose group you want
limit:int=25, # members to return
columns:list=None, # store columns to return per member row
min_count:int=None, # usearch: smallest cluster to emit
max_count:int=None, # usearch: largest cluster to emit
k:int=8, # neighbours per node in the kNN fallback
dtype:type=float16
):The group key belongs to — its family, not a ranked list of what is nearest to it.
def clusters(
min_count:int=None, # usearch: smallest cluster to emit
max_count:int=None, # usearch: largest cluster to emit
k:int=8, # neighbours per node in the kNN fallback
min_size:int=2, # drop groups smaller than this
label_k:int=4, # terms per c-TF-IDF label
members:int=24, # member rows fetched per cluster
columns:list=None, # store columns to return per member row
max_label_chars:int=4000, # per-member text budget for labelling
dtype:type=float16
):The corpus grouped by embedding shape, each group named by c-TF-IDF.
# clusters(): two well-separated blobs -> two groups, each labelled by its own vocabulary
from litesearch.core import database
_cdb = database()
_cst = _cdb.get_store('cl', ann=True, ndim=8, metric='cosine')
_rs = np.random.RandomState(0)
_a = np.concatenate([np.ones((30,4)), np.zeros((30,4))], 1) + _rs.randn(30,8)*0.05
_b = np.concatenate([np.zeros((30,4)), np.ones((30,4))], 1) + _rs.randn(30,8)*0.05
_txt = [f'kernel gradient tensor sample {i}' for i in range(30)] + [f'invoice ledger payment row {i}' for i in range(30)]
_cst.insert_all([{'content':t,'embedding':v.astype(np.float16).tobytes()}
for t,v in zip(_txt, np.concatenate([_a,_b]))])
assert _cst.rebuild_index() == 60
_cl = _cst.clusters(min_size=3)
print(_cl.note, '|', [(c.size, c.label) for c in _cl.clusters][:4])
assert len(_cl.clusters) >= 2, _cl.note
assert _cl.method in ('usearch','knn')
_labels = ' '.join(c.label for c in _cl.clusters)
assert 'tensor' in _labels or 'kernel' in _labels, _labels # c-TF-IDF names a group after what only it says
assert all(r['rowid'] in c.member_keys for c in _cl.clusters for r in c.members)
# peers(): every member of a group has the rest of that group as its family
_c0 = first(_cl.clusters, lambda c: 'kernel' in c.label or 'tensor' in c.label)
_pr = _cst.peers(_c0.centroid, limit=5, columns=['content'])
print(_pr.note, '|', [h['content'][:24] for h in _pr.hits])
assert _pr.hits and _pr.method == _cl.method, _pr.note
assert all('kernel' in h['content'] for h in _pr.hits), _pr.hits
# a row the clustering left on its own still gets an answer -- with the fallback named in `note`
_solo = first(range(1, 61), lambda r: not _cst.peers(r).method == _cl.method)
if _solo: assert 'nearest neighbours' in _cst.peers(_solo).note
# degradation is reported, never silent
_tiny = database().get_store('tiny', ann=True, ndim=4, metric='cosine')
_tiny.insert_all([{'content':'x','embedding':np.zeros(4,dtype=np.float16).tobytes()}])
_tiny.rebuild_index()
assert _tiny.clusters().clusters == [] and 'too few' in _tiny.clusters().note
assert 'nearest neighbours' in _tiny.peers(1).note
assert database().get_store('plain').clusters().note.endswith('is not an ANN store')
assert database().get_store('empty', ann=True).clusters().note.endswith('has no vectors yet')6 clusters over 60 vectors (knn) | [(8, 'payment, ledger, row, invoice'), (8, 'sample, kernel, tensor, gradient'), (7, 'sample, kernel, tensor, gradient'), (5, 'payment, ledger, row, invoice')]
cluster of 8 (knn) | ['kernel gradient tensor s', 'kernel gradient tensor s', 'kernel gradient tensor s', 'kernel gradient tensor s', 'kernel gradient tensor s']