Some documents in a store are noise: they match many queries and answer none. A scraped cookie banner, a navigation menu or a licence footer shows up in search results because its words are common, and it pushes real answers down the list. This module gives every document a noise score so you can find such documents and decide whether to remove them.
The score needs no labelled examples, no extra model and no second pass over the source files. It is computed from the chunk vectors and texts the store already holds. A cookie banner gives itself away in several ways at once: near-identical copies of it appear on many pages, its vector sits away from most of the corpus, and its words are ones that occur everywhere.
The default weights were measured in vishalakshi (evals/noise.py and evals/RESULTS.md in that repository) on three test corpora with boilerplate documents planted in them. The weighted score separated boilerplate from real documents with a mean AUC of 0.988 (0.967 to 1.000). AUC is the chance that a randomly chosen noise document scores above a randomly chosen real one, so 0.5 is a coin toss and 1.0 is perfect.
The functions read a tree store, the kind Index and add_doc build, because they need each chunk’s doc_id and the docs table for titles. A plain get_store table has neither.
chunk_matrix reads every chunk that has an embedding and returns its id, its document id, its text and its vector, with the vectors stacked into one float32 array and scaled to unit length. With unit vectors, a dot product is the cosine similarity. Pass limit to work on a random sample of chunks in a large store, and dtype if the store holds vectors in a width other than float16.
Return (ids, doc_ids, texts, V) for every chunk with an embedding, with V the unit-length vectors as one float32 array.
These helpers compute the quantities the features are built from. _knn finds each chunk’s nearest other chunks: exactly, by matrix product, up to 30,000 chunks, and through a temporary HNSW index above that, unless usearch’s HNSW index failed its probe. _centroids finds topic centres with k-means. chunk_idf splits text into words with TOKEN_RE and scores how rare each word is across chunks: inverse document frequency (IDF) is high for a word few chunks use and near zero for a word in nearly every chunk. _entropy measures how evenly a row of probabilities is spread, from 0 (all on one value) to 1 (spread evenly).
Return the inverse document frequency of every word, counting each chunk as one document.
noise_features computes nine numbers per document. Each is computed per chunk and averaged over the document’s chunks, and in each a larger value means more likely noise.
| feature | what it measures | default weight |
|---|---|---|
hub |
how often the document’s chunks appear in other chunks’ nearest-neighbour lists, as a z-score | 1.0 |
spread_chunk |
how evenly a chunk’s similarity is spread over the topic centres, so a chunk that belongs to no topic scores high | 0.8 |
dup_out |
share of chunks with a near-duplicate (similarity at least tau, 0.9) in another document |
0.8 |
off_centre |
cosine distance from the mean vector of the whole store | 0.5 |
promiscuity |
share of all logged queries that retrieved the document | 0.5 |
low_idf |
average IDF of the chunk’s words, negated, so common words score high | 0.3 |
redundancy |
share of chunks with a near-duplicate in the same document | 0.1 |
spread_doc |
how evenly the document’s chunks spread over topics | 0 |
short |
share of chunks under 200 characters | 0 |
spread_doc and short are computed but weighted 0, because on their own they scored below chance in vishalakshi’s measurements. spread_doc in particular marks survey documents that cover many topics, which are not noise.
promiscuity is the only feature that needs data from outside the store: a log of which queries retrieved which documents, passed as seen, a dict from doc id to the set of queries. Without it the feature is 0 for every document and the other eight decide the score.
noise_scores turns the features into one score per document and returns every document, most likely noise first, with its title and each feature value. Each feature is first converted to its rank among the documents and rescaled to about -1.7 to 1.7 by robust_z, so one extreme value cannot dominate. The score is then the weighted average using NOISE_W, or your own weights, or a fitted Ranker. Ranks matter here: with plain z-scores, one outlier document on hub dropped the measured AUC from 0.988 to 0.55.
def noise_scores(
db, # an open Database
store:str='store', # the chunk table
weights:dict=None, # per-feature blend; None -> `NOISE_W`
ranker:NoneType=None, # a fitted `Ranker` instead of a fixed blend
prefix:str=None, # the tree prefix, for the docs table; None -> from `store`
**kw
)->fastcore.foundation.L: # forwarded to `noise_features`Score every document for noise and return them most likely noise first, with their feature values.
Replace each feature column by its rank, rescaled to about -1.7 to 1.7, with tied values sharing a rank.
def noise_features(
db, # an open Database
store:str='store', # the chunk table
seen:dict=None, # doc_id -> the set of queries it was retrieved for, if logged
k:int=10, # neighbours per chunk
topics:int=64, # topic centroids for the spread features
tau:float=0.9, # similarity floor for "this is a duplicate"
temp:float=8.0, # softmax temperature for chunk-to-topic spread
limit:int=None, # sample this many chunks instead of using all of them
exact_max:int=30000, # above this many chunks, neighbours come from HNSW instead of an exact search
seed:int=0, # random seed for sampling and k-means
dtype:type=float16, # the width the store holds
)->fastcore.basics.AttrDict:Compute the nine noise features for every document in a tree store, as AttrDict(doc_ids, X, names).
The fixed weights work without any labels. If you have marked some documents as noise and some as keepers, Ranker learns weights from those marks instead; vishalakshi measured 0.996 AUC that way, from six marked documents and eighteen keepers.
Ranker is a pairwise learning-to-rank model. You give it rows of features, a group for each row, and a target value. Within each group, every pair of rows where one has the higher target becomes a training example, and the model learns weights that score the higher row above the lower one. It is a logistic regression on the feature differences of those pairs, with an L2 penalty, fitted by Newton’s method (iteratively reweighted least squares, IRLS).
For noise, put every document in one group and use 1 for noise and 0 for keep, then pass the fitted ranker to noise_scores(ranker=...). vishalakshi also fits one on its log of search feedback, with one group per query, to re-rank results. weights() shows what the model learned, and to_dict and from_dict save and restore it as plain JSON-compatible data.
A linear model that learns, from labelled pairs of rows, which rows should rank higher.
The tests build an Index with five real documents and three near-copies of a cookie banner. They use hash_embed, a hashing encoder that needs no model or network. It matches spellings only, so it separates the banner less cleanly than a real encoder would.
from fastcore.test import test_eq, test_close
from litesearch import Index, hash_embed
class HashEmbed:
"`hash_embed` behind an `.encode`, so these tests need no model and no network."
def __init__(self, dims=256): self.dims = dims
def encode(self, xs, **kw): return hash_embed(list(xs), ndim=self.dims)