# quality
No labels, no model, no second pass over the corpus. The features are
document-level aggregates of quantities the chunk store already holds.
The blend of them scores 0.988 AUC against hand-marked junk.
The vector legs answer “what is this like”. These answer “is this worth
retrieving at all”. A cookie banner is a near-duplicate of every other
page’s, sits far from the corpus mean, and is made of words that are
everywhere.
## The vectors, as one array
------------------------------------------------------------------------
source
### Database.chunk_matrix
``` python
def chunk_matrix(
store:str='store', # the chunk table
limit:int=None, # sample this many chunks instead of using all of them
seed:int=0, dtype:type=float16, # the width the store holds
)->tuple:
```
*`(ids, doc_ids, texts, V)`: every chunk’s stored vector, L2-normalised,
as one array.*
## Nearest neighbours, centroids and the rest of the arithmetic
------------------------------------------------------------------------
source
### chunk_idf
``` python
def chunk_idf(
texts
)->dict:
```
*Inverse document frequency over chunks: boilerplate is made of words
that are everywhere.*
## The features
Nine of them, all oriented larger-is-noisier. `promiscuity` is the only
one needing anything outside the store. It counts how many separate
questions found a document, which a caller passes in as `seen` if it
keeps a log. Without one the feature is zero and the others carry the
score.
------------------------------------------------------------------------
source
### noise_scores
``` python
def noise_scores(
db, # an open Database
store:str='store', # the chunk table
weights:dict=None, # per-feature blend; None -> `NOISE_W`
ranker:NoneType=None, # a fitted `Ranker` instead of a fixed blend
prefix:str=None, # the tree prefix, for the docs table; None -> from `store`
**kw
)->L: # forwarded to `noise_features`
```
*Every document, most suspicious first, with the features that put it
there.*
------------------------------------------------------------------------
source
### noise_features
``` python
def noise_features(
db, # an open Database
store:str='store', # the chunk table
seen:dict=None, # doc_id -> the set of queries it was retrieved for, if logged
k:int=10, # neighbours per chunk
topics:int=64, # topic centroids for the spread features
tau:float=0.9, # similarity floor for "this is a duplicate"
temp:float=8.0, # softmax temperature for chunk-to-topic spread
limit:int=None, # sample this many chunks instead of using all of them
exact_max:int=30000, seed:int=0, dtype:type=float16, # the width the store holds
)->AttrDict:
```
*Per-document noise features, computed from the vectors already in the
file.*
## The ranker
A logistic model over standardised features, fitted by IRLS with an L2
penalty. Two callers use it. One scores noise over the features above.
The other re-ranks retrieval features pairwise. Both are the same twelve
lines of arithmetic.
------------------------------------------------------------------------
source
### Ranker
``` python
def Ranker(
names, w:NoneType=None, mu:NoneType=None, sd:NoneType=None, meta:NoneType=None
):
```
*Pairwise linear learning-to-rank over the feedback log.*
## Tests
``` python
from fastcore.test import test_eq, test_close
from litesearch import Index, hash_embed
class HashEmbed:
"`hash_embed` behind an `.encode`, so these tests need no model and no network."
def __init__(self, dims=256): self.dims = dims
def encode(self, xs, **kw): return hash_embed(list(xs), ndim=self.dims)
```