# everything in this notebook, from the command line
!python -m evals.ingest_bench --helpingestion performance
Where ingestion time goes, and what the 0.1.16 fixes changed
What this measures
The retrieval evals ask what litesearch finds. This one asks what it costs to put anything in: parsing, chunking, embedding, SQLite upserts, ANN index maintenance, graph extraction. It exists because a corpus of a million documents fails on the write side long before it fails on the read side, and nothing in the retrieval evals would have caught it.
Everything here is driven by evals/ingest_bench.py and runs on the same evals.corpus genres the retrieval evals use, repartitioned into as many documents as a run needs. hash_embed stands in for an encoder so the numbers are litesearch’s own overhead rather than ONNX inference, pass --encoder potion to put a real model back in.
Read the exponent, not the throughput. Per-document cost is only a cost if it is constant. Every size sweep below reports the slope of log(time) on log(n): 1.0 is linear, and anything approaching 2.0 does not reach a million documents at any hardware budget. The single most important number in this notebook is that add_dir went from 1.72 to 0.91.
Cells are eval: false because a full sweep takes tens of minutes. The recorded output of each is in the markdown beneath it, measured on 4 CPUs with SQLite in WAL mode.
The headline
| before 0.1.16 | after | ||
|---|---|---|---|
add_dir, 400 markdown docs |
112.2s, exponent 1.72 | 9.2s, exponent 0.91 | 12.2x |
| code ingestion, 2,169 files | 190.4s | 7.3s | 26x |
process_content |
469–527 rows/s | 912 rows/s | 1.8x |
resolve_entities, 8,487 entities |
13.3s, exponent 1.35 | 5.8s, exponent 1.08 | 2.3x |
The multipliers are the least interesting part. The exponents are the finding: ingest used to get slower per document as the corpus grew, and now it does not.
1. Document ingestion, swept by corpus size
add_doc used to end with rebuild_index(), which reads every embedding blob in the store and reconstructs the whole HNSW graph. Per document that is O(corpus), so a directory walk was O(N²), to build an index nobody reads until ingestion finishes.
It now mirrors only that document’s keys into the index, the way sync always has, and add_dir treats the walk as a bulk load: FTS triggers suspended for the duration, one index rebuild at the end.
from evals.ingest_bench import bench_docs
bench_docs(sizes=(25,50,100,200,400))before (0.1.15) after (0.1.16)
add_dir(25) 1.64s 65.5 ms/doc add_dir(25) 0.80s 32.1 ms/doc
add_dir(50) 3.31s 66.3 ms/doc add_dir(50) 1.21s 24.1 ms/doc
add_dir(100) 8.35s 83.5 ms/doc add_dir(100) 2.16s 21.6 ms/doc
add_dir(200) 29.05s 145.2 ms/doc add_dir(200) 5.06s 25.3 ms/doc
add_dir(400) 112.18s 267.8 ms/doc add_dir(400) 9.21s 23.0 ms/doc
exponent 1.72 exponent 0.91Per-document cost used to climb 4x from 25 to 400 documents. It is now flat.
The profile of the old 200-document run is where this came from: usearch.compiled.add_many was 17.4s of 34.7s, half the wall clock, alongside 409,730 row fetches and 407,928 np.frombuffer calls, all of them the quadratic re-reads.
# the isolating experiment: same corpus, same final index, rebuild deferred to once per batch.
# run against 0.1.15 this reports the gap the fix closed.
from evals.ingest_bench import bench_deferred_index
bench_deferred_index(sizes=(50,100,200,400))On 0.1.15 this was 1.35x at 50 documents rising to 7.01x at 400, the gap grows linearly with corpus size, which is what “unbounded” means here. On 0.1.16 both arms are the fixed path, so it now measures only the cost of the final rebuild (0.49s of 16.0s at 400 documents).
Benchmark on an idle machine. A concurrent nbdev-test (2 workers on 4 CPUs) was enough to turn the 50-document case into 55.9s against 3.7s, while leaving every vector count identical.
2. Writes: one transaction, and the FTS triggers
process_content called insert_all outside any transaction, so apswutils committed per chunk and WAL fsynced each time. The batching already existed, but behind parallel=True, which is a concurrency flag, not a throughput one. It now always batches; parallel controls only the widened busy timeout, which is all it ever meant.
The remaining gap between a store with FTS and one without is the per-row trigger. bulk_load suspends the triggers and uses FTS5’s own rebuild once at the end.
from evals.ingest_bench import bench_store, verify_bulk_load
bench_store(sizes=(2000,8000,32000))
verify_bulk_load()8,000 rows 0.1.15 0.1.16
no fts, autocommit 4.56s
no fts, one txn 0.50s 9.1x
fts, autocommit 7.44s
fts, one txn 1.46s 5.1x
process_content(parallel=True) 1.11s the win, gated behind a flag
process_content end to end 527/s 913/s exponent 1.04 -> 1.00bulk_load is 2.05x, and populates the index rather than skipping it:
bulk_load=False 1.15s rows=8000 fts_rows=8000 hits(vessel)=37
bulk_load=True 0.66s rows=8000 fts_rows=8000 hits(vessel)=37Only worth it for a load: rebuild is O(rows in the table), so wrapping a two-row update in it is strictly slower than letting the triggers run.
3. Code files
Two independent problems, both worth more than they look.
ast.get_source_segment re-splits the entire source file on every call, and pyparse called it once per chunk it emitted, O(top-level defs x file size) per file. Profiling 400 files of transformers, ast._splitlines_no_ff was 47.3s of 70.0s. pyparse also walked the whole tree tagging every node with a parent, then filtered on “parent is the Module”, which is what tree.body already means.
And dir2chunks/pkg2chunks ran file_parse on a thread pool. It is ast.parse plus a pure-python walk and holds the GIL end to end, so the pool contended rather than overlapped.
# byte-for-byte against the implementation this replaced, over a real package
from evals.ingest_bench import verify_pyparse
verify_pyparse('litesearch') # or any large tree: site-packages/transformersOver 2,169 files of transformers:
old new chunks mismatches
default 159.80s 22.36s 7.1x 14,444 both 0
imports=True, assigns=True 281.89s 24.66s 11.4x 32,720 both 0Zero mismatches on content and metadata. nbs/02_data.ipynb pins _seg against ast.get_source_segment on the cases where the two line-splitters actually differ: a decorated def, a one-line class, a form feed, and a non-ascii line before the node. An earlier draft that used str.splitlines had 9 mismatches for exactly that reason, form feeds are breaks for splitlines and deliberately not for _splitlines_no_ff.
The pool, same 2,169 files:
serial (n_workers=0) 17.67s
threads (the old default) 24.00s 0.76x, slower than serial
processes (new default) 7.34s 3.3x190.4s on the old default against 7.34s: 26x end to end. Directories below MIN_PARALLEL_FILES (64) stay serial, because below that a pool costs more to start than the parse it saves.
4. PDFs
pdf_parse is Rust (pdf-oxide) but holds the GIL, so it threads no better than the AST parser does.
from evals.ingest_bench import bench_pdf
bench_pdf()8 corpus PDFs, 489 pages
serial 4.07s 120.2 pages/s
thread(4) 4.41s 111.0 pages/s 0.92x
process(4) 2.14s 228.3 pages/s 1.90x
thread(8) 3.92s 124.8 pages/s 1.04x
process(8) 2.68s 182.7 pages/s 1.52xSingle-document rates for reference: 120 pages/s on the text-heavy directives, 34 pages/s on an image-heavy arXiv paper.
Still open: add_dir walks files serially. _parse_files shows the shape of the fix but it has not been applied to the document path, because PDF parsing and SQLite writing want different pool sizes.
5. Graph
Three separate problems, found by profiling rather than by reading: throughput, memory, and a recall cliff that no timing would ever have shown.
Throughput. insert_chunk was 9.3s of a 36.1s build, build_graph wrote entities, mentions and edges through insert_all directly, outside any transaction, so it had the autocommit-per-batch problem process_content had just been fixed for. One write_txn around the writes took 1,000 chunks from 21.6s to 11.5s.
Cores. With the writes fixed, extraction is 88% of a build (26.6s of 30.3s), and it is a pure function of the chunk text, so the build is a map with a cheap serial reduce after it. n_workers= spreads it over processes; threads are useless because yake and spaCy are both python and both hold the GIL. spaCy’s own n_process is used rather than reinvented, since it forks around the pipeline instead of shipping one per task. Below MIN_PARALLEL_CHUNKS (200) a drain stays serial.
Order is preserved by both pools and the reduce depends on it: ents.setdefault keeps the kind of an entity’s first mention, so a reordered stream would relabel entities.
Memory. build_graph accumulated entities, mentions, edges and every co-occurrence window for the whole call. Windows are the one term that never saturates: entity vocabulary follows Heaps’ law and flattens, mentions can be flushed, but a window is one per sentence forever. Peak RSS grew straight-line with the corpus, so the ceiling was the machine.
Three changes, each verified to leave the graph byte-identical:
batch=flushes mentions per batch and moves windows to a scratch SQLite table._pmi_edgesalready walked its windows twice, because the hub cut-off is not known until the first pass ends, so anything with__len__and a re-iterable__iter__substitutes for the list.- Pair counting moved into SQL. It was the last unbounded structure: pairs are quadratic in window size, so the counter tracked the corpus even after the windows stopped being held. SQLite groups on a temp b-tree instead of in RSS.
for c in L(chunks)became a plain iteration.L()materialises, so a caller who passed a generator specifically to keep a corpus off the heap had it read into a list before the first chunk was touched.
Recall. _lexical_pairs skipped any shared-token block bigger than max_group. Blocks grow with the corpus, so the pairs that stopped being proposed were exactly the ones a larger corpus added, a resolution that quietly gets worse the more you feed it, with nothing in any timing to show why. Oversized blocks are now windowed on sorted names rather than skipped: sorting puts containment variants adjacent (“polonium” beside “polonium isotope”), which is the case the lexical pass exists for, and the work stays O(block x max_group).
from evals.ingest_bench import bench_graph, bench_lexical_recall, bench_graph_memory
bench_graph(sizes=(500,1000,2000))
bench_graph(sizes=(1000,2000), n_workers=4) # extraction on a process pool
bench_lexical_recall(sizes=(250,500,1000,2000)) # exhaustive O(E^2) ground truth
bench_graph_memory(sizes=(2000,4000,8000)) # run per-size in its own process for a clean RSSThroughput. The transaction fix and the pool are independent, and they compound:
1,000 chunks yake spaCy
before (0.1.16) ~45 ch/s
+ write_txn 86 ch/s 21 ch/s
+ 2 workers 141 ch/s
+ 4 workers 207 ch/s 39 ch/s
4.6x total 1.87x from the poolresolve_entities after the batched probe:
before after
resolve n=500 3.61s 1.67s
resolve n=1000 7.43s 3.28s
resolve n=2000 13.26s 5.62s
resolve exponent 1.35 1.08Memory, generator input with batch=500 against a list with no batching, each in its own process:
chunks list, no batch generator + batch marginal KB/chunk (batched)
2000 113 MB 82 MB
4000 171 MB 109 MB 13.5
8000 281 MB 150 MB 10.3
16000, 190 MB 5.0The marginal cost halves as the corpus doubles, it now tracks entity vocabulary, which saturates, rather than corpus size, which does not. That is the difference between a corpus that fits and one that does not.
Lexical blocking recall, against an exhaustive _lex_ok ground truth:
entities truth pairs skipping (before) windowing (after)
1,377 1,569 97.0% 99.8%
3,254 3,713 94.4% 99.5%
5,650 8,193 86.1% 98.3%
8,487 15,004 75.9% 97.2%At 8,487 entities the old blocking was proposing three quarters of the valid merges and falling. Windowing costs 2.2x the pairs examined and no measurable time, resolve_entities at n=2000 is 5.62s against 5.65s, because _toks is cached and _uf_union short-circuits pairs already merged. It finds more: 6,422 merges against 5,896.
An honest note on determinism
resolve_entities does not return the same answer twice. Rebuilt from scratch at n=2000 it gives merged=5896 on one run and merged=5891 on the next, about 0.1%.
This is not the batching. usearch builds its HNSW graph across threads, so two rebuilds of the same vectors are two slightly different graphs, and an approximate probe over a different graph returns slightly different neighbours. It was true before this change too. The check that separates the two holds the index still and varies only the probe:
from evals.ingest_bench import verify_ann_probe, verify_resolve
verify_ann_probe(n=2000) # one fixed index, batched vs the per-entity loop
verify_resolve(sizes=(1000,2000), reps=2) # full rebuilds: shows the ~0.1% drift== ann probe equivalence (8487 entities, fixed index, k=8) ==
looped x3: 67896 pairs, stable: True
batched x3: 67896 pairs, stable: True
identical: True symmetric difference: 0Given the same index, the two probes agree exactly and both are deterministic. So the batching is equivalent, and the drift belongs to index construction, worth knowing if you ever diff two graph builds and expect them to match.
verify_resolve reports a partition hash rather than the id→canon map, for the same reason: _uf_union breaks rank ties by arrival order, so which member of a merged group ends up canonical is not stable even when the grouping is.
Considered and set aside: sharding
One database per profile was measured and is not worth doing for performance.
Before the fixes it bought 4.7x on ingest at 8 shards, but only because it divided the quadratic. The fix alone on a single database (19.8s over 400 documents) beat 8-way sharding without it (23.8s), and with ingest linear, sharding adds ~12% and then flattens.
On reads it is single-digit milliseconds in both directions at this corpus size: fanning out to 16 shards costs 34.9ms against 8.0ms for one index, and routing to a known shard saves 6ms. Neither is a reason to restructure anything.
bench_shards and bench_shard_reads remain in evals/ingest_bench.py for the one argument that survives, a resident HNSW index that no longer fits in memory, which is about RAM, not speed.
What is still open at 10⁶
- Embedding, deliberately excluded from every number here. With litesearch’s own overhead out of the way, the encoder is now the dominant cost, and it is the part that wants a GPU or a process pool.
add_dirwalks files serially, parsing parallelises 1.9x on 4 cores and the document path does not do it yet, because PDF parsing and SQLite writing want different pool sizes.build_graphscales to cores, not across machines. 207 chunks/s on 4 cores puts a million chunks at ~1.3 hours. The reduce is serial, so more cores keep helping until it dominates.n_workerswith a smallbatchpays pool startup per drain, the queue drains once per batch, so the two interact: 4.87s unbatched against 5.33s atbatch=500.build_graphmemory is sublinear, not constant. What remains is entity vocabulary and the edge dict, both of which grow with a saturating curve rather than a flat one. Fine to 10⁶ on the measured slope; worth re-measuring before 10⁷.- spaCy is slower than the yake fallback (9.5 vs 16.4 chunks/s under tracemalloc) while extracting ~1.7x the entities. Which is the better trade has not been measured on retrieval quality, only on cost.