# everything in this notebook, from the command line
!python -m evals.ingest_bench --helpingestion performance
Where ingestion time goes, and what the 0.1.16 fixes changed
What this measures
The retrieval evals measure what litesearch finds. This one measures what it costs to put documents in: parsing, chunking, embedding, SQLite writes, keeping the vector index up to date, and building an entity graph. A corpus of a million documents fails on the write side long before it fails on the read side, and the retrieval evals would never have shown that.
Everything here runs through evals/ingest_bench.py, on the same evals.corpus documents the retrieval evals use, split into as many documents as a run needs. hash_embed, a model-free hashing embedder from litesearch.utils, stands in for the encoder, so the timings are litesearch’s own overhead and not model inference. Pass --encoder potion to put a real model back in.
Read the exponent, not the throughput. A cost per document only means something if it stays constant as the corpus grows. Each sweep over corpus sizes reports the slope of log(time) against log(number of documents): 1.0 means time grows in proportion to the corpus, and a slope near 2.0 means doubling the corpus quadruples the time, which never reaches a million documents on any hardware. The most important number in this notebook is that add_dir went from 1.72 to 0.91.
The cells are eval: false because a full sweep takes tens of minutes. The markdown under each cell records its output, measured on 4 CPUs with SQLite in WAL mode, for versions 0.1.15 and 0.1.16.
Three things have changed since. The graph code (build_graph, resolve_entities) moved to vruksha, and the graph benchmarks import it from there. add_dir now parses files on a process pool once at least four of them are slow to parse (PDF, XML, HTML, notebooks), which closes the open item at the end of section 4. And process_content(parallel=...) no longer does anything; give the connection a wait with database(busy_timeout=...) instead.
The headline
| before 0.1.16 | after | ||
|---|---|---|---|
add_dir, 400 markdown docs |
112.2s, exponent 1.72 | 9.2s, exponent 0.91 | 12.2x |
| code ingestion, 2,169 files | 190.4s | 7.3s | 26x |
process_content |
469 to 527 rows/s | 912 rows/s | 1.8x |
resolve_entities, 8,487 entities |
13.3s, exponent 1.35 | 5.8s, exponent 1.08 | 2.3x |
The speedups matter less than the exponents. Ingestion used to get slower per document as the corpus grew, and now it does not.
1. Adding documents, at growing corpus sizes
HNSW is the graph-based index usearch uses for approximate nearest-neighbour search over the embeddings. add_doc used to finish with rebuild_index(), which reads every embedding in the store and builds the whole HNSW graph again. That costs time in proportion to the corpus for each document added, so adding a directory took time proportional to the square of its size, all to build an index nobody reads until ingestion is over.
add_doc now adds only the new document’s vectors to the index, as sync always did. add_dir treats the whole walk as a bulk load: it suspends the FTS triggers while it writes, and rebuilds the vector index once at the end.
from evals.ingest_bench import bench_docs
bench_docs(sizes=(25,50,100,200,400))before (0.1.15) after (0.1.16)
add_dir(25) 1.64s 65.5 ms/doc add_dir(25) 0.80s 32.1 ms/doc
add_dir(50) 3.31s 66.3 ms/doc add_dir(50) 1.21s 24.1 ms/doc
add_dir(100) 8.35s 83.5 ms/doc add_dir(100) 2.16s 21.6 ms/doc
add_dir(200) 29.05s 145.2 ms/doc add_dir(200) 5.06s 25.3 ms/doc
add_dir(400) 112.18s 267.8 ms/doc add_dir(400) 9.21s 23.0 ms/doc
exponent 1.72 exponent 0.91The cost per document used to grow fourfold between 25 and 400 documents. It is now flat.
The profile of the old 200-document run showed where the time went. usearch.compiled.add_many took 17.4s of 34.7s, half the run, alongside 409,730 row fetches and 407,928 np.frombuffer calls, all of them re-reading vectors that were already indexed.
# the isolating experiment: same corpus, same final index, rebuild deferred to once per batch.
# run against 0.1.15 this reports the gap the fix closed.
from evals.ingest_bench import bench_deferred_index
bench_deferred_index(sizes=(50,100,200,400))On 0.1.15 the deferred rebuild was 1.35x faster at 50 documents and 7.01x faster at 400, a gap that grows with the corpus. On 0.1.16 both arms take the fixed path, so the benchmark now measures only the cost of the final rebuild: 0.49s of 16.0s at 400 documents.
Benchmark on an idle machine. A concurrent nbdev-test run, 2 workers on 4 CPUs, was enough to turn the 50-document case into 55.9s against 3.7s, with every vector count unchanged.
2. Writes: one transaction, and the FTS triggers
process_content called insert_all outside any transaction, so each chunk was committed separately and SQLite’s write-ahead log was flushed to disk each time. Batching in one transaction already existed, but only behind parallel=True, a flag that was meant to be about concurrency. process_content now always writes in transactions of chunk rows. (Since then parallel has become a no-op.)
The remaining gap between a store with full-text search and one without comes from the triggers that update the FTS index on every row. bulk_load removes those triggers for the duration of a load and then rebuilds the FTS index once with FTS5’s rebuild command.
from evals.ingest_bench import bench_store, verify_bulk_load
bench_store(sizes=(2000,8000,32000))
verify_bulk_load()8,000 rows 0.1.15 0.1.16
no fts, autocommit 4.56s
no fts, one txn 0.50s 9.1x
fts, autocommit 7.44s
fts, one txn 1.46s 5.1x
process_content(parallel=True) 1.11s the gain, then only behind a flag
process_content end to end 527/s 913/s exponent 1.04 -> 1.00bulk_load is 2.05x faster and still fills the FTS index, as the same row count and the same hits show:
bulk_load=False 1.15s rows=8000 fts_rows=8000 hits(vessel)=37
bulk_load=True 0.66s rows=8000 fts_rows=8000 hits(vessel)=37Use it only for a load. rebuild reads every row in the table, so wrapping a two-row update in bulk_load is slower than letting the triggers run.
3. Code files
Two separate problems, each costing more than it looked.
ast.get_source_segment splits the entire source file into lines on every call, and pyparse called it once for each chunk it produced, so the cost grew with the number of top-level definitions times the size of the file. Over 400 files of transformers, ast._splitlines_no_ff took 47.3s of 70.0s. pyparse also walked the whole syntax tree to tag each node with its parent and then kept the nodes whose parent was the module, which is exactly the list tree.body already holds.
dir2chunks and pkg2chunks also ran file_parse on a thread pool. Parsing is ast.parse plus a pure-Python walk and holds Python’s global interpreter lock (GIL) throughout, so the threads queued behind each other instead of running together.
# byte-for-byte against the implementation this replaced, over a real package
from evals.ingest_bench import verify_pyparse
verify_pyparse('litesearch') # or any large tree: site-packages/transformersOver 2,169 files of transformers:
old new chunks mismatches
default 159.80s 22.36s 7.1x 14,444 both 0
imports=True, assigns=True 281.89s 24.66s 11.4x 32,720 both 0The new parser produced the same content and metadata for every chunk. nbs/02_data.ipynb tests _seg against ast.get_source_segment on the cases where the two ways of splitting lines differ: a decorated function, a one-line class, a form feed, and a non-ASCII line before the node. An earlier draft that used str.splitlines had 9 mismatches for exactly that reason: str.splitlines breaks lines at a form feed and ast._splitlines_no_ff does not.
The pool, on the same 2,169 files:
serial (n_workers=0) 17.67s
threads (the old default) 24.00s 0.76x, slower than serial
processes (new default) 7.34s 3.3xFrom 190.4s on the old default to 7.34s, 26x end to end. A directory with fewer than MIN_PARALLEL_FILES (64) files is parsed serially, because below that the pool costs more to start than it saves.
4. PDFs
pdf_parse runs in Rust (pdf-oxide) but does not release the GIL, so threads help it no more than they help the Python parser.
from evals.ingest_bench import bench_pdf
bench_pdf()8 corpus PDFs, 489 pages
serial 4.07s 120.2 pages/s
thread(4) 4.41s 111.0 pages/s 0.92x
process(4) 2.14s 228.3 pages/s 1.90x
thread(8) 3.92s 124.8 pages/s 1.04x
process(8) 2.68s 182.7 pages/s 1.52xFor a single document: 120 pages/s on the text-heavy directives, 34 pages/s on an arXiv paper full of images.
Open when this was measured: add_dir parsed files one at a time. It now parses on a process pool once at least four files need a slow parse (see the note at the top).
5. Graph
The entity graph links chunks through the people, places and terms they mention. Its builder now lives in vruksha, but these fixes were measured here. Profiling found three problems: throughput, memory, and a loss of recall that no timing would have shown.
Throughput. insert_chunk took 9.3s of a 36.1s build. build_graph wrote entities, mentions and edges with insert_all outside any transaction, the same commit-per-batch problem just fixed in process_content. One write_txn around the writes took 1,000 chunks from 21.6s to 11.5s.
Cores. With the writes fixed, extracting entities was 88% of a build (26.6s of 30.3s). Extraction depends only on each chunk’s text, so the build is a parallel map followed by a cheap serial merge. n_workers= runs the map on processes; threads do not help, because yake and spaCy are Python code that holds the GIL. For spaCy the builder uses spaCy’s own n_process, which forks once around the loaded pipeline instead of sending the pipeline to every task. A batch of fewer than MIN_PARALLEL_CHUNKS (200) chunks runs serially.
Both pools keep chunks in order, and the merge relies on that: ents.setdefault keeps the kind recorded at an entity’s first mention, so a different order would relabel entities.
Memory. build_graph held every entity, mention, edge and co-occurrence window for the whole call. A co-occurrence window is a span of text, one per sentence, in which two entities count as appearing together. Windows were the term that never levels off. The number of distinct entities grows ever more slowly (Heaps’ law) and mentions can be written out as they go, but windows keep coming at one per sentence. Peak memory grew in a straight line with the corpus, so the machine’s memory set the limit.
Three changes fixed it, and each was checked to produce a byte-identical graph:
batch=writes mentions out after each batch and moves windows into a scratch SQLite table._pmi_edgesalready read its windows twice, because the cut-off for over-connected entities is only known after the first pass, so any object with__len__and a repeatable__iter__can stand in for the list.- Pair counting moved into SQL. It was the last structure without a bound: the number of pairs grows with the square of the window size, so the counter kept growing with the corpus even after the windows were no longer held in memory. SQLite groups the pairs in a temporary B-tree on disk instead.
for c in L(chunks)became a plain loop.L()builds a list, so a caller who passed a generator to keep the corpus out of memory had it read into a list before the first chunk was processed.
Recall. Entity resolution merges different spellings of the same entity, and _lexical_pairs proposes candidate pairs from groups of names that share a token. It skipped any group larger than max_group. Groups grow with the corpus, so the skipped pairs were exactly the ones a larger corpus added, and resolution got quietly worse the more text it saw. Large groups are now handled in windows over the sorted names instead of being skipped. Sorting puts variants that contain each other side by side (“polonium” beside “polonium isotope”), which is the case the lexical pass exists for, and the work stays proportional to the group size times max_group.
from evals.ingest_bench import bench_graph, bench_lexical_recall, bench_graph_memory
bench_graph(sizes=(500,1000,2000))
bench_graph(sizes=(1000,2000), n_workers=4) # extraction on a process pool
bench_lexical_recall(sizes=(250,500,1000,2000)) # exhaustive O(E^2) ground truth
bench_graph_memory(sizes=(2000,4000,8000)) # run per-size in its own process for a clean RSSThroughput. The transaction fix and the pool are independent, and their gains multiply:
1,000 chunks yake spaCy
before (0.1.16) ~45 ch/s
+ write_txn 86 ch/s 21 ch/s
+ 2 workers 141 ch/s
+ 4 workers 207 ch/s 39 ch/s
4.6x total 1.87x from the poolresolve_entities after batching its index probes:
before after
resolve n=500 3.61s 1.67s
resolve n=1000 7.43s 3.28s
resolve n=2000 13.26s 5.62s
resolve exponent 1.35 1.08Peak memory, a generator with batch=500 against a list with no batching, each run in its own process:
chunks list, no batch generator + batch marginal KB/chunk (batched)
2000 113 MB 82 MB
4000 171 MB 109 MB 13.5
8000 281 MB 150 MB 10.3
16000 (not run) 190 MB 5.0The extra memory per chunk halves each time the corpus doubles. It now follows the number of distinct entities, which levels off, rather than the size of the corpus, which does not.
Recall of lexical blocking, against an exhaustive check of every pair with _lex_ok:
entities truth pairs skipping (before) windowing (after)
1,377 1,569 97.0% 99.8%
3,254 3,713 94.4% 99.5%
5,650 8,193 86.1% 98.3%
8,487 15,004 75.9% 97.2%At 8,487 entities the old blocking proposed only three quarters of the valid merges, and the share was falling. Windowing examines 2.2 times as many pairs and costs no measurable time (resolve_entities at n=2000 takes 5.62s against 5.65s), because _toks is cached and _uf_union skips pairs that are already merged. It finds more merges: 6,422 against 5,896.
Determinism
resolve_entities does not give the same answer twice. Rebuilt from scratch at n=2000, it reported merged=5896 on one run and merged=5891 on the next, a difference of about 0.1%.
The batching is not the cause. usearch builds its HNSW graph on several threads, so two builds from the same vectors give slightly different graphs, and an approximate search over a different graph returns slightly different neighbours. This was true before the change as well. The check that tells the two causes apart keeps the index fixed and varies only the probe:
from evals.ingest_bench import verify_ann_probe, verify_resolve
verify_ann_probe(n=2000) # one fixed index, batched vs the per-entity loop
verify_resolve(sizes=(1000,2000), reps=2) # full rebuilds: shows the ~0.1% drift== ann probe equivalence (8487 entities, fixed index, k=8) ==
looped x3: 67896 pairs, stable: True
batched x3: 67896 pairs, stable: True
identical: True symmetric difference: 0On the same index the two probes agree exactly and each is deterministic. The batching is therefore equivalent, and the drift comes from building the index. Expect it if you compare two graph builds.
verify_resolve reports a hash of the partition, which entities ended up grouped together, rather than of the map from each entity to its canonical name. _uf_union breaks ties by arrival order, so which member of a group becomes canonical can change even when the groups do not.
Open at a million documents
These were the open items when this was measured.
- Embedding is left out of every number here. With litesearch’s own overhead out of the way, the encoder is now the largest cost, and it is the part that wants a GPU or a process pool.
add_dirparsed files one at a time. It now uses a process pool for slow-to-parse files.build_graphuses more cores, not more machines. At 207 chunks/s on 4 cores, a million chunks takes about 1.3 hours. The merge is serial, so more cores help until it becomes the bottleneck.n_workerswith a smallbatchpays the pool’s start-up cost once per batch: 4.87s without batching against 5.33s atbatch=500.build_graphmemory grows more slowly than the corpus but still grows. What remains is the entity vocabulary and the edge dict, both of which level off rather than stay flat. The measured slope is fine to a million documents; measure again before ten million.- spaCy extracts about 1.7 times as many entities as the yake fallback but is slower (9.5 against 16.4 chunks/s, measured under tracemalloc). Which is the better trade for retrieval quality has not been measured.