# topics ## Schema Three tables beside the chunk store. Entities live in a regular `get_store` so they inherit the existing ANN machinery, resolution reuses `ann_search`/`rebuild_index` rather than adding a second index path. ------------------------------------------------------------------------ source ### Database.get_graph ``` python def get_graph( store:str='store', # chunk store the graph is built over prefix:str=None, # table prefix (default: '' for 'store', else '_') ann:bool=True, # register an ANN index on the entity store (needed for resolution) **kw ): ``` *Create the entity/mention/edge tables for a chunk store. Idempotent; returns the tables.* ## Topics from the ANN index usearch clusters off the HNSW graph but walks index levels, so it raises `Index too small to cluster!` on a small corpus. [`_knn_clusters`](https://Karthik777.github.io/litesearch/topics.html#_knn_clusters) falls back to a greedy pass over the kNN graph, which works at any size. Labels are c-TF-IDF: term frequency inside a cluster, weighted by inverse document frequency across clusters. Plain frequency names every cluster after the same few words. [`_knn_clusters`](https://Karthik777.github.io/litesearch/topics.html#_knn_clusters) seeds at the densest unassigned node and claims its unassigned neighbours, so clusters stay bounded. Label propagation collapses a dense kNN graph into one component. ------------------------------------------------------------------------ source ### topic_nodes ``` python def topic_nodes( db, # Database store:str='store', # chunk store (must be ANN-registered) prefix:NoneType=None, # graph table prefix min_count:NoneType=None, # usearch cluster min size max_count:NoneType=None, # usearch cluster max size k:int=8, # neighbours per node in the fallback kNN graph min_size:int=2, # smallest cluster kept as a topic label_k:int=4, # terms per topic label max_label_chars:int=4000, dtype:type=float16 ): ``` *Cluster the store index into topic nodes labelled by c-TF-IDF. Returns {topics, method}.* ------------------------------------------------------------------------ source ### ctfidf_labels ``` python def ctfidf_labels( texts, # one text per member, aligned with `lab` lab, # cluster index per member k, # number of clusters top_n:int=4, # terms per label stop:NoneType=None, # stopword set (defaults to _STOP) sep:str=', ' ): ``` *Name each cluster by terms common inside it and rare across the other clusters.* ## Clusters and peers `store.clusters()` maps the corpus into labelled groups. `store.peers(rowid)` returns the group one row belongs to. `peers` is not `ann_neighbors`. k-NN returns 15 nearest things whether or not they are related; a cluster returns the family. `peers` falls back to `ann_neighbors` when the index cannot be clustered, and says so in `note`. Both return `note`, because an empty clustering and a broken index look identical to a caller. `clusters` returns `AttrDict(clusters, method, note)`, each cluster `AttrDict(centroid, size, label, member_keys, members)`. `method` is `usearch` or `knn`. `peers` returns `AttrDict(hits, method, note)`. ------------------------------------------------------------------------ source ### Table.peers ``` python def peers( key:int, # usearch key (rowid) whose group you want limit:int=25, # members to return columns:list=None, # store columns to return per member row min_count:int=None, # usearch: smallest cluster to emit max_count:int=None, # usearch: largest cluster to emit k:int=8, # neighbours per node in the kNN fallback dtype:type=float16 ): ``` *The group `key` belongs to — its family, not a ranked list of what is nearest to it.* ------------------------------------------------------------------------ source ### Table.clusters ``` python def clusters( min_count:int=None, # usearch: smallest cluster to emit max_count:int=None, # usearch: largest cluster to emit k:int=8, # neighbours per node in the kNN fallback min_size:int=2, # drop groups smaller than this label_k:int=4, # terms per c-TF-IDF label members:int=24, # member rows fetched per cluster columns:list=None, # store columns to return per member row max_label_chars:int=4000, # per-member text budget for labelling dtype:type=float16 ): ``` *The corpus grouped by embedding shape, each group named by c-TF-IDF.* ``` python # clusters(): two well-separated blobs -> two groups, each labelled by its own vocabulary from litesearch.core import database _cdb = database() _cst = _cdb.get_store('cl', ann=True, ndim=8, metric='cosine') _rs = np.random.RandomState(0) _a = np.concatenate([np.ones((30,4)), np.zeros((30,4))], 1) + _rs.randn(30,8)*0.05 _b = np.concatenate([np.zeros((30,4)), np.ones((30,4))], 1) + _rs.randn(30,8)*0.05 _txt = [f'kernel gradient tensor sample {i}' for i in range(30)] + [f'invoice ledger payment row {i}' for i in range(30)] _cst.insert_all([{'content':t,'embedding':v.astype(np.float16).tobytes()} for t,v in zip(_txt, np.concatenate([_a,_b]))]) assert _cst.rebuild_index() == 60 _cl = _cst.clusters(min_size=3) print(_cl.note, '|', [(c.size, c.label) for c in _cl.clusters][:4]) assert len(_cl.clusters) >= 2, _cl.note assert _cl.method in ('usearch','knn') _labels = ' '.join(c.label for c in _cl.clusters) assert 'tensor' in _labels or 'kernel' in _labels, _labels # c-TF-IDF names a group after what only it says assert all(r['rowid'] in c.member_keys for c in _cl.clusters for r in c.members) # peers(): every member of a group has the rest of that group as its family _c0 = first(_cl.clusters, lambda c: 'kernel' in c.label or 'tensor' in c.label) _pr = _cst.peers(_c0.centroid, limit=5, columns=['content']) print(_pr.note, '|', [h['content'][:24] for h in _pr.hits]) assert _pr.hits and _pr.method == _cl.method, _pr.note assert all('kernel' in h['content'] for h in _pr.hits), _pr.hits # a row the clustering left on its own still gets an answer -- with the fallback named in `note` _solo = first(range(1, 61), lambda r: not _cst.peers(r).method == _cl.method) if _solo: assert 'nearest neighbours' in _cst.peers(_solo).note # degradation is reported, never silent _tiny = database().get_store('tiny', ann=True, ndim=4, metric='cosine') _tiny.insert_all([{'content':'x','embedding':np.zeros(4,dtype=np.float16).tobytes()}]) _tiny.rebuild_index() assert _tiny.clusters().clusters == [] and 'too few' in _tiny.clusters().note assert 'nearest neighbours' in _tiny.peers(1).note assert database().get_store('plain').clusters().note.endswith('is not an ANN store') assert database().get_store('empty', ann=True).clusters().note.endswith('has no vectors yet') ``` 6 clusters over 60 vectors (knn) | [(8, 'payment, ledger, row, invoice'), (8, 'sample, kernel, tensor, gradient'), (7, 'sample, kernel, tensor, gradient'), (5, 'payment, ledger, row, invoice')] cluster of 8 (knn) | ['kernel gradient tensor s', 'kernel gradient tensor s', 'kernel gradient tensor s', 'kernel gradient tensor s', 'kernel gradient tensor s']