# ingestion performance ### What this measures The retrieval evals ask what litesearch finds. This one asks what it costs to put anything in: parsing, chunking, embedding, SQLite upserts, ANN index maintenance, graph extraction. It exists because a corpus of a million documents fails on the write side long before it fails on the read side, and nothing in the retrieval evals would have caught it. Everything here is driven by `evals/ingest_bench.py` and runs on the same `evals.corpus` genres the retrieval evals use, repartitioned into as many documents as a run needs. [`hash_embed`](https://Karthik777.github.io/litesearch/utils.html#hash_embed) stands in for an encoder so the numbers are litesearch’s own overhead rather than ONNX inference, pass `--encoder potion` to put a real model back in. **Read the exponent, not the throughput.** Per-document cost is only a cost if it is constant. Every size sweep below reports the slope of log(time) on log(n): 1.0 is linear, and anything approaching 2.0 does not reach a million documents at any hardware budget. The single most important number in this notebook is that `add_dir` went from 1.72 to 0.91. Cells are `eval: false` because a full sweep takes tens of minutes. The recorded output of each is in the markdown beneath it, measured on 4 CPUs with SQLite in WAL mode. ``` python # everything in this notebook, from the command line !python -m evals.ingest_bench --help ``` ### The headline
before 0.1.16 after
add_dir, 400 markdown docs 112.2s, exponent 1.72 9.2s, exponent 0.91 12.2x
code ingestion, 2,169 files 190.4s 7.3s 26x
process_content 469–527 rows/s 912 rows/s 1.8x
resolve_entities, 8,487 entities 13.3s, exponent 1.35 5.8s, exponent 1.08 2.3x
The multipliers are the least interesting part. The exponents are the finding: ingest used to get slower per document as the corpus grew, and now it does not. ### 1. Document ingestion, swept by corpus size `add_doc` used to end with `rebuild_index()`, which reads every embedding blob in the store and reconstructs the whole HNSW graph. Per document that is O(corpus), so a directory walk was O(N²), to build an index nobody reads until ingestion finishes. It now mirrors only that document’s keys into the index, the way `sync` always has, and `add_dir` treats the walk as a bulk load: FTS triggers suspended for the duration, one index rebuild at the end. ``` python from evals.ingest_bench import bench_docs bench_docs(sizes=(25,50,100,200,400)) ``` before (0.1.15) after (0.1.16) add_dir(25) 1.64s 65.5 ms/doc add_dir(25) 0.80s 32.1 ms/doc add_dir(50) 3.31s 66.3 ms/doc add_dir(50) 1.21s 24.1 ms/doc add_dir(100) 8.35s 83.5 ms/doc add_dir(100) 2.16s 21.6 ms/doc add_dir(200) 29.05s 145.2 ms/doc add_dir(200) 5.06s 25.3 ms/doc add_dir(400) 112.18s 267.8 ms/doc add_dir(400) 9.21s 23.0 ms/doc exponent 1.72 exponent 0.91 Per-document cost used to climb 4x from 25 to 400 documents. It is now flat. The profile of the old 200-document run is where this came from: `usearch.compiled.add_many` was 17.4s of 34.7s, half the wall clock, alongside 409,730 row fetches and 407,928 `np.frombuffer` calls, all of them the quadratic re-reads. ``` python # the isolating experiment: same corpus, same final index, rebuild deferred to once per batch. # run against 0.1.15 this reports the gap the fix closed. from evals.ingest_bench import bench_deferred_index bench_deferred_index(sizes=(50,100,200,400)) ``` On 0.1.15 this was 1.35x at 50 documents rising to 7.01x at 400, the gap grows linearly with corpus size, which is what “unbounded” means here. On 0.1.16 both arms are the fixed path, so it now measures only the cost of the final rebuild (0.49s of 16.0s at 400 documents). **Benchmark on an idle machine.** A concurrent `nbdev-test` (2 workers on 4 CPUs) was enough to turn the 50-document case into 55.9s against 3.7s, while leaving every vector count identical. ### 2. Writes: one transaction, and the FTS triggers [`process_content`](https://Karthik777.github.io/litesearch/core.html#process_content) called `insert_all` outside any transaction, so apswutils committed per chunk and WAL fsynced each time. The batching already existed, but behind `parallel=True`, which is a concurrency flag, not a throughput one. It now always batches; `parallel` controls only the widened busy timeout, which is all it ever meant. The remaining gap between a store with FTS and one without is the per-row trigger. `bulk_load` suspends the triggers and uses FTS5’s own `rebuild` once at the end. ``` python from evals.ingest_bench import bench_store, verify_bulk_load bench_store(sizes=(2000,8000,32000)) verify_bulk_load() ``` 8,000 rows 0.1.15 0.1.16 no fts, autocommit 4.56s no fts, one txn 0.50s 9.1x fts, autocommit 7.44s fts, one txn 1.46s 5.1x process_content(parallel=True) 1.11s the win, gated behind a flag process_content end to end 527/s 913/s exponent 1.04 -> 1.00 `bulk_load` is 2.05x, and populates the index rather than skipping it: bulk_load=False 1.15s rows=8000 fts_rows=8000 hits(vessel)=37 bulk_load=True 0.66s rows=8000 fts_rows=8000 hits(vessel)=37 Only worth it for a load: `rebuild` is O(rows in the table), so wrapping a two-row update in it is strictly slower than letting the triggers run. ### 3. Code files Two independent problems, both worth more than they look. `ast.get_source_segment` re-splits the entire source file on every call, and [`pyparse`](https://Karthik777.github.io/litesearch/data.html#pyparse) called it once per chunk it emitted, O(top-level defs x file size) per file. Profiling 400 files of `transformers`, `ast._splitlines_no_ff` was **47.3s of 70.0s**. [`pyparse`](https://Karthik777.github.io/litesearch/data.html#pyparse) also walked the whole tree tagging every node with a `parent`, then filtered on “parent is the Module”, which is what `tree.body` already means. And [`dir2chunks`](https://Karthik777.github.io/litesearch/data.html#dir2chunks)/[`pkg2chunks`](https://Karthik777.github.io/litesearch/data.html#pkg2chunks) ran [`file_parse`](https://Karthik777.github.io/litesearch/data.html#file_parse) on a thread pool. It is `ast.parse` plus a pure-python walk and holds the GIL end to end, so the pool contended rather than overlapped. ``` python # byte-for-byte against the implementation this replaced, over a real package from evals.ingest_bench import verify_pyparse verify_pyparse('litesearch') # or any large tree: site-packages/transformers ``` Over 2,169 files of `transformers`: old new chunks mismatches default 159.80s 22.36s 7.1x 14,444 both 0 imports=True, assigns=True 281.89s 24.66s 11.4x 32,720 both 0 Zero mismatches on content and metadata. `nbs/02_data.ipynb` pins [`_seg`](https://Karthik777.github.io/litesearch/data.html#_seg) against `ast.get_source_segment` on the cases where the two line-splitters actually differ: a decorated def, a one-line class, a form feed, and a non-ascii line before the node. An earlier draft that used `str.splitlines` had 9 mismatches for exactly that reason, form feeds are breaks for `splitlines` and deliberately not for `_splitlines_no_ff`. The pool, same 2,169 files: serial (n_workers=0) 17.67s threads (the old default) 24.00s 0.76x, slower than serial processes (new default) 7.34s 3.3x 190.4s on the old default against 7.34s: **26x** end to end. Directories below `MIN_PARALLEL_FILES` (64) stay serial, because below that a pool costs more to start than the parse it saves. ### 4. PDFs [`pdf_parse`](https://Karthik777.github.io/litesearch/data.html#pdf_parse) is Rust (pdf-oxide) but holds the GIL, so it threads no better than the AST parser does. ``` python from evals.ingest_bench import bench_pdf bench_pdf() ``` 8 corpus PDFs, 489 pages serial 4.07s 120.2 pages/s thread(4) 4.41s 111.0 pages/s 0.92x process(4) 2.14s 228.3 pages/s 1.90x thread(8) 3.92s 124.8 pages/s 1.04x process(8) 2.68s 182.7 pages/s 1.52x Single-document rates for reference: 120 pages/s on the text-heavy directives, 34 pages/s on an image-heavy arXiv paper. **Still open:** `add_dir` walks files serially. [`_parse_files`](https://Karthik777.github.io/litesearch/data.html#_parse_files) shows the shape of the fix but it has not been applied to the document path, because PDF parsing and SQLite writing want different pool sizes. ### 5. Graph Three separate problems, found by profiling rather than by reading: throughput, memory, and a recall cliff that no timing would ever have shown. **Throughput.** `insert_chunk` was 9.3s of a 36.1s build, `build_graph` wrote entities, mentions and edges through `insert_all` directly, outside any transaction, so it had the autocommit-per-batch problem [`process_content`](https://Karthik777.github.io/litesearch/core.html#process_content) had just been fixed for. One [`write_txn`](https://Karthik777.github.io/litesearch/core.html#write_txn) around the writes took 1,000 chunks from 21.6s to 11.5s. **Cores.** With the writes fixed, extraction is 88% of a build (26.6s of 30.3s), and it is a pure function of the chunk text, so the build is a map with a cheap serial reduce after it. `n_workers=` spreads it over processes; threads are useless because yake and spaCy are both python and both hold the GIL. spaCy’s own `n_process` is used rather than reinvented, since it forks around the pipeline instead of shipping one per task. Below `MIN_PARALLEL_CHUNKS` (200) a drain stays serial. Order is preserved by both pools and the reduce depends on it: `ents.setdefault` keeps the `kind` of an entity’s first mention, so a reordered stream would relabel entities. **Memory.** `build_graph` accumulated entities, mentions, edges and every co-occurrence window for the whole call. Windows are the one term that never saturates: entity vocabulary follows Heaps’ law and flattens, mentions can be flushed, but a window is one per sentence forever. Peak RSS grew straight-line with the corpus, so the ceiling was the machine. Three changes, each verified to leave the graph byte-identical: - `batch=` flushes mentions per batch and moves windows to a scratch SQLite table. `_pmi_edges` already walked its windows twice, because the hub cut-off is not known until the first pass ends, so anything with `__len__` and a re-iterable `__iter__` substitutes for the list. - Pair counting moved into SQL. It was the last unbounded structure: pairs are quadratic in window size, so the counter tracked the corpus even after the windows stopped being held. SQLite groups on a temp b-tree instead of in RSS. - `for c in L(chunks)` became a plain iteration. `L()` materialises, so a caller who passed a generator specifically to keep a corpus off the heap had it read into a list before the first chunk was touched. **Recall.** `_lexical_pairs` skipped any shared-token block bigger than `max_group`. Blocks grow with the corpus, so the pairs that stopped being proposed were exactly the ones a larger corpus added, a resolution that quietly gets worse the more you feed it, with nothing in any timing to show why. Oversized blocks are now windowed on sorted names rather than skipped: sorting puts containment variants adjacent (“polonium” beside “polonium isotope”), which is the case the lexical pass exists for, and the work stays O(block x max_group). ``` python from evals.ingest_bench import bench_graph, bench_lexical_recall, bench_graph_memory bench_graph(sizes=(500,1000,2000)) bench_graph(sizes=(1000,2000), n_workers=4) # extraction on a process pool bench_lexical_recall(sizes=(250,500,1000,2000)) # exhaustive O(E^2) ground truth bench_graph_memory(sizes=(2000,4000,8000)) # run per-size in its own process for a clean RSS ``` Throughput. The transaction fix and the pool are independent, and they compound: 1,000 chunks yake spaCy before (0.1.16) ~45 ch/s + write_txn 86 ch/s 21 ch/s + 2 workers 141 ch/s + 4 workers 207 ch/s 39 ch/s 4.6x total 1.87x from the pool `resolve_entities` after the batched probe: before after resolve n=500 3.61s 1.67s resolve n=1000 7.43s 3.28s resolve n=2000 13.26s 5.62s resolve exponent 1.35 1.08 Memory, generator input with `batch=500` against a list with no batching, each in its own process: chunks list, no batch generator + batch marginal KB/chunk (batched) 2000 113 MB 82 MB 4000 171 MB 109 MB 13.5 8000 281 MB 150 MB 10.3 16000, 190 MB 5.0 The marginal cost halves as the corpus doubles, it now tracks entity vocabulary, which saturates, rather than corpus size, which does not. That is the difference between a corpus that fits and one that does not. Lexical blocking recall, against an exhaustive `_lex_ok` ground truth: entities truth pairs skipping (before) windowing (after) 1,377 1,569 97.0% 99.8% 3,254 3,713 94.4% 99.5% 5,650 8,193 86.1% 98.3% 8,487 15,004 75.9% 97.2% At 8,487 entities the old blocking was proposing three quarters of the valid merges and falling. Windowing costs 2.2x the pairs examined and **no measurable time**, `resolve_entities` at n=2000 is 5.62s against 5.65s, because `_toks` is cached and `_uf_union` short-circuits pairs already merged. It finds more: 6,422 merges against 5,896. #### An honest note on determinism `resolve_entities` does not return the same answer twice. Rebuilt from scratch at n=2000 it gives `merged=5896` on one run and `merged=5891` on the next, about 0.1%. This is not the batching. usearch builds its HNSW graph across threads, so two rebuilds of the same vectors are two slightly different graphs, and an approximate probe over a different graph returns slightly different neighbours. It was true before this change too. The check that separates the two holds the index still and varies only the probe: ``` python from evals.ingest_bench import verify_ann_probe, verify_resolve verify_ann_probe(n=2000) # one fixed index, batched vs the per-entity loop verify_resolve(sizes=(1000,2000), reps=2) # full rebuilds: shows the ~0.1% drift ``` == ann probe equivalence (8487 entities, fixed index, k=8) == looped x3: 67896 pairs, stable: True batched x3: 67896 pairs, stable: True identical: True symmetric difference: 0 Given the same index, the two probes agree exactly and both are deterministic. So the batching is equivalent, and the drift belongs to index construction, worth knowing if you ever diff two graph builds and expect them to match. `verify_resolve` reports a **partition** hash rather than the id→canon map, for the same reason: `_uf_union` breaks rank ties by arrival order, so which member of a merged group ends up canonical is not stable even when the grouping is. ### Considered and set aside: sharding One database per profile was measured and is not worth doing for performance. Before the fixes it bought 4.7x on ingest at 8 shards, but only because it divided the quadratic. The fix alone on a single database (19.8s over 400 documents) beat 8-way sharding without it (23.8s), and with ingest linear, sharding adds ~12% and then flattens. On reads it is single-digit milliseconds in both directions at this corpus size: fanning out to 16 shards costs 34.9ms against 8.0ms for one index, and routing to a known shard saves 6ms. Neither is a reason to restructure anything. `bench_shards` and `bench_shard_reads` remain in `evals/ingest_bench.py` for the one argument that survives, a resident HNSW index that no longer fits in memory, which is about RAM, not speed. ### What is still open at 10⁶ - **Embedding**, deliberately excluded from every number here. With litesearch’s own overhead out of the way, the encoder is now the dominant cost, and it is the part that wants a GPU or a process pool. - **`add_dir` walks files serially**, parsing parallelises 1.9x on 4 cores and the document path does not do it yet, because PDF parsing and SQLite writing want different pool sizes. - **`build_graph` scales to cores, not across machines.** 207 chunks/s on 4 cores puts a million chunks at ~1.3 hours. The reduce is serial, so more cores keep helping until it dominates. - **`n_workers` with a small `batch` pays pool startup per drain**, the queue drains once per batch, so the two interact: 4.87s unbatched against 5.33s at `batch=500`. - **`build_graph` memory is sublinear, not constant.** What remains is entity vocabulary and the edge dict, both of which grow with a saturating curve rather than a flat one. Fine to 10⁶ on the measured slope; worth re-measuring before 10⁷. - **spaCy is slower than the yake fallback** (9.5 vs 16.4 chunks/s under tracemalloc) while extracting ~1.7x the entities. Which is the better trade has not been measured on retrieval quality, only on cost.