data

Turn PDFs, text and code into chunks ready to index, and turn a user’s question into an FTS5 query

Before a document can be searched it has to be cut into pieces. A search returns pieces, not whole files: a 15-page paper should come back as the paragraph that answers the question, not as page 1. Each piece is called a chunk, and the object that decides where to cut is a chunker.

This page holds the tools that read files and produce chunks, plus the functions that clean up a query before it reaches SQLite’s full-text search. Most applications never call them directly, because Index.add and add_dir in litesearch.tree call them for you. Reach for them when you build your own pipeline on database(), or when you want to see what a file turns into before you index it.

The sections, in order: reading PDFs, chunking text, reading code and other files, profiles for special document kinds, listing installed packages, preparing queries, and building a PDF from images.

Reading PDFs

pdf_oxide opens and reads PDFs. litesearch adds a few methods to its PdfDocument class, each taking an optional page range st to end (0-based, end excluded, all pages by default):

method	returns
pdf_texts	the plain text of each page
pdf_markdown	each page as markdown, with headings and tables marked up
pdf_links	every hyperlink in the document, as a list of URLs
pdf_images	metadata for every image, or with output_dir, saves the images and returns each page’s text with links to them
pdf_tables	every table found, with its rows, cells and position
pdf_spans	runs of text with their font size, weight and position on the page

These are thin wrappers over pdf_oxide and do no cleaning. For indexing, use pdf_parse below instead.

---

source

PdfDocument.pdf_spans

def pdf_spans(
    doc:pdf_oxide.pdf_oxide.PdfDocument, st:int=0, end:NoneType=None
)->fastcore.foundation.L:

Every run of text on the pages, with its font size, weight and bounding box.

---

source

PdfDocument.pdf_tables

def pdf_tables(
    doc:pdf_oxide.pdf_oxide.PdfDocument, st:int=0, end:NoneType=None
)->fastcore.foundation.L:

Every table on the pages, with its rows, cells and bounding box.

---

source

PdfDocument.pdf_markdown

def pdf_markdown(
    doc:pdf_oxide.pdf_oxide.PdfDocument, st:int=0, end:NoneType=None, **kw
)->fastcore.foundation.L:

Each page as markdown, with headings and tables marked up.

---

source

PdfDocument.pdf_images

def pdf_images(
    doc:pdf_oxide.pdf_oxide.PdfDocument, st:int=0, end:NoneType=None, output_dir:NoneType=None
)->fastcore.foundation.L:

Metadata for every image on the pages, or, when output_dir is given, each page as text with its images saved there.

---

source

PdfDocument.pdf_links

def pdf_links(
    doc:pdf_oxide.pdf_oxide.PdfDocument, st:int=0, end:NoneType=None
)->fastcore.foundation.L:

Every hyperlink on the pages from st to end, as a list of URLs.

---

source

PdfDocument.pdf_texts

def pdf_texts(
    doc:pdf_oxide.pdf_oxide.PdfDocument, st:int=0, end:NoneType=None
)->fastcore.foundation.L:

The plain text of each page from st to end.

pdf_parse is the function to use when you index a PDF. It returns a list with one markdown string per page. The work is done by pdflite, a separate package whose helpers are re-exported here. pdflite converts each page with pdf_oxide, joins words hyphenated across a line break, and collapses the extra spaces and blank lines the conversion leaves. Some pages come out with their columns scrambled, so that a figure such as a price ends up on a line of its own, away from its label. For those pages pdflite tries the plain-text layer instead and keeps whichever version strands fewer values.

A scanned book has no text layer at all, only page images, and needs OCR (optical character recognition, reading the text from the images). ocr_selection controls it. With 'auto', the default, OCR runs over the whole document when pdf_oxide reports any page as scanned. 'on' always runs it and 'off' never does. OCR needs the liteparse package, which is not installed with litesearch.

Images on the pages are saved to out_path/image_dir, and the markdown links to them by that relative path. With no out_path, they go to pdfs/images under the current folder. preserve_layout=True keeps indentation and runs of spaces, which tables laid out with spaces need.

Index and add_dir call pdf_parse and keep its page boundaries. The document tree and the default chunk size were both measured on this one-string-per-page split.

---

source

pdf_parse

def pdf_parse(
    pdf:pdf_oxide.pdf_oxide.PdfDocument | str | pathlib.Path | bytes,
    out_path:str | pathlib.Path=None, # folder that holds the image folder (default `pdfs`); image links in the markdown are relative to it
    image_dir:str='images', # name of the folder, inside `out_path`, that extracted images are saved to
    preserve_layout:bool=False, # keep indentation and runs of spaces, which space-aligned tables need
    ocr_selection:str='auto', # 'auto' runs OCR when pdf_oxide reports a scanned page; 'on' always, 'off' never
)->list:

A PDF as a list of markdown strings, one per page, with OCR for scanned documents.

mds=pdf_parse(Path('pdfs/attention_is_all_you_need.pdf'))

Dictionary used where Stream expected, treating as empty stream
Dictionary used where Stream expected, treating as empty stream
Dictionary used where Stream expected, treating as empty stream
Dictionary used where Stream expected, treating as empty stream

Chunking text

The default chunker is chonkie’s FastChunker, set to 512 bytes. It walks through the text and cuts each piece at the last newline, full stop or question mark before the 512-byte limit, so a chunk usually ends at the end of a line or sentence. Where there is none, it cuts at the limit. 512 bytes is about 500 characters of English, or about 170 characters of Devanagari, which takes three bytes per letter. evals/ measured 512-byte chunks against page-sized ones, and the smaller chunks found the right section more often.

SafeFastChunker

FastChunker cuts by byte position, so a cut can land in the middle of a character that takes more than one byte, such as an em dash, a curly quote or any Devanagari letter. It then raises UnicodeDecodeError, and a long unbroken paragraph with one such character, which most scraped web pages contain, is enough to trigger it. SafeFastChunker moves each cut back to the start of the character it split. No text is lost; a cut moves by at most three bytes. litesearch uses it everywhere in place of FastChunker.

---

source

getattr

def __getattr__(
    nm
):

from chonkie import FastChunker
SafeFastChunker = _safe_chunker()
assert SafeFastChunker is _safe_chunker() and issubclass(SafeFastChunker, FastChunker)

Importing litesearch does not import chonkie, model2vec, tokenizers or codesigs. Importing chonkie alone takes about 190 ms, and a process that only searches an existing store never needs any of them. Each is imported the first time a function that needs it runs. The test below checks this in a fresh Python process.

import subprocess, sys
_heavy = ['chonkie', 'chonkie_core', 'model2vec', 'tokenizers', 'codesigs']
_chk = f"""import sys, litesearch
from litesearch import correct, fts_or, fts_phrases, rank_fts, rrf_all, doc_encoder, query_encoder, static_embedder
print([m for m in {_heavy!r} if m in sys.modules])"""
assert subprocess.run([sys.executable, '-c', _chk], capture_output=True, text=True, check=True).stdout.strip() == '[]'
_chk = "import sys, litesearch; litesearch.chunk_markdown('a b'); print(litesearch.SafeFastChunker.__name__, 'chonkie' in sys.modules)"
assert subprocess.run([sys.executable, '-c', _chk], capture_output=True, text=True, check=True).stdout.strip() == 'SafeFastChunker True'

---

source

PdfDocument.pdf_chunks

def pdf_chunks(
    *, chunker:BaseChunker=None, # any chonkie chunker; None uses a 512-byte SafeFastChunker
)->fastcore.foundation.L: # passed to chunk_markdown, so only `chunker`

Every chunk of the PDF as a (page, chunk_idx, text) tuple, with pages counted from 1.

---

source

chunk_markdown

def chunk_markdown(
    text:str, # markdown text (e.g. one page from pdf_parse())
    chunker:BaseChunker=None, # any chonkie chunker; None uses a 512-byte SafeFastChunker
)->fastcore.foundation.L:

Split text into chunks of at most 512 bytes, or with chunker if given, and return their text.

_pdf2 = next((p for p in ['pdfs/attention_is_all_you_need.pdf', 'nbs/pdfs/attention_is_all_you_need.pdf'] if Path(p).exists()), None)
if _pdf2:
    _doc = PdfDocument(_pdf2)
    _chunks = _doc.pdf_chunks()
    assert len(_chunks) > 0
    assert all(len(t) == 3 for t in _chunks), 'each chunk must be (page, chunk_idx, text)'
    print(f'pdf_chunks: {len(_chunks)} chunks from {_doc.page_count()} pages (sample: {repr(_chunks[0][2][:60])})...')
else:
    print('Skipping chunk test — PDF not found')

Dictionary used where Stream expected, treating as empty stream
Dictionary used where Stream expected, treating as empty stream
Dictionary used where Stream expected, treating as empty stream
Dictionary used where Stream expected, treating as empty stream

pdf_chunks: 98 chunks from 15 pages (sample: 'Provided proper attribution is provided, Google hereby grant')...

There are three ways to chunk text:

• chunk_markdown(text) returns the chunks as a list of strings. Use it when you only need the text.
• chunk_spans(text) returns (start, end, text) for each chunk, as character positions in the string you passed in. Use it when you need to map a chunk back to its place in the source, for example to highlight a hit inside the full document.
• PdfDocument.pdf_chunks() runs pdf_parse and then chunk_markdown on each page, and returns (page, chunk_idx, text). Pages are counted from 1 and chunk_idx restarts at 0 on each page. Use it to index a PDF page by page without building a document tree.

All three take an optional chunker, any chonkie chunker, if 512 bytes does not suit your text.

---

source

chunk_spans

def chunk_spans(
    text:str, # text to split
    chunker:BaseChunker=None, # any chonkie chunker; None uses a 512-byte SafeFastChunker
)->fastcore.foundation.L:

Split text into chunks and return each as (start, end, text), where text[start:end] gives the chunk back.

_t = "First sentence here. Second sentence follows. Third one ends it."
_spans = chunk_spans(_t)
assert len(_spans) >= 1
for s,e,txt in _spans:
    assert _t[s:e] == txt          # offsets index back into the original text exactly
assert _spans[0][0] == 0           # first chunk starts at 0

for _bad in ['a'*510 + '—' + 'b'*600,     # the cut lands mid-character
             'a'*400 + '—'*200,           # every candidate cut is mid-character
             '—'*700]:
    assert ''.join(chunk_markdown(_bad)) == _bad, 'text lost or altered by chunking'
    for s, e, txt in chunk_spans(_bad): assert _bad[s:e] == txt

# the upstream chunker on the same input is what used to reach `add_doc`
try: FastChunker(chunk_size=512)('a'*510 + '—' + 'b'*600); assert False, 'chonkie fixed it upstream'
except UnicodeDecodeError: pass

# a paragraph split still happens where it should, and ASCII text chunks exactly as before
_para = ('x'*200 + '\n\n')*3
assert chunk_markdown(_para, SafeFastChunker(chunk_size=512, delimiters='\n\n')) == \
       chunk_markdown(_para, FastChunker(chunk_size=512, delimiters='\n\n'))

Reading code and other files

Code needs different chunks from prose. A 512-byte cut through the middle of a function is useless as a search result, so code is cut at definitions instead: each top-level function or class becomes one chunk. Every chunk is a dict with the text under content and facts about where it came from under metadata, which is the shape get_store and sync in litesearch.core take.

file_parse(path) picks a reader by the file’s extension:

extension	reader	one chunk per
.py	pyparse	top-level function or class, plus imports or assignments if asked
.ipynb	ipynb_parse	notebook cell, of any type
.js, .ts, .go, .rs and the rest of code_exts	non_py_sigs	signature, with the body replaced by {...}
.pdf	PdfDocument.pdf_chunks	512-byte chunk
.md	chunk_markdown	512-byte chunk
.txt	chunk_texts	512-byte chunk, cut at blank lines where possible

Any other extension gives an empty list. For languages other than Python only the signatures are indexed, through the codesigs package, so a search finds the function by its name and parameters but not by what its body does.

To read many files at once, pkg2chunks reads every file of an installed package and dir2chunks every file under a folder. Both are further down.

---

source

chunk_texts

def chunk_texts(
    text:str
):

Split plain text into chunks of at most 512 bytes, cutting at blank lines where possible.

---

source

non_py_sigs

def non_py_sigs(
    p
):

One chunk per function or class signature in a non-Python code file, read with codesigs.

---

source

ipynb_parse

def ipynb_parse(
    p
):

One chunk per cell of a notebook, with the cell type in its metadata.

---

source

pyparse

def pyparse(
    p:pathlib.Path=None, # path to a python file
    code:str=None, # code string to parse
    imports:bool=False, # also return each `from ... import ...` statement as a chunk
    assigns:bool=False, # also return each top-level assignment as a chunk
)->fastcore.foundation.L:

One chunk per top-level function and class in a Python file or code string, each a dict of content and metadata.

---

source

spec

def spec(
    pkg:str, # import or distribution name, or requirement string
    path:list=None, # site-packages folders to search; None searches the running interpreter
)->_frozen_importlib.ModuleSpec | None:

The importlib ModuleSpec for a package, found by import name or distribution name, or None.

---

source

pkg_dist

def pkg_dist(
    pkg:str, # package name or requirement string
    path:list=None, # site-packages folders to search; None searches the running interpreter
):

The installed distribution for a package name or requirement string, or None.

---

source

repo_root

def repo_root()->pathlib.Path:

The root folder of the git repository holding the current folder, or None outside one.

Profiles: how to read one kind of document

Some documents need more than a different chunker. A Sanskrit text in TEI XML, for example, needs a reader that knows an <lg xml:id=...> element is a verse, a tree that treats a citation as the start of a chapter, and a chunker that cuts at the daṇḍa (।), the mark that ends a half-verse. A Profile bundles all of these for one kind of document and gives them a name. When add_file or add_dir in litesearch.tree reads a file, it asks profile_for whether a registered profile claims it. If one does, the profile’s parts are used; if none does, the file goes through the default pipeline above.

Profile

The fields, all optional except name:

field	what it is
exts	the extensions the profile claims, comma-separated, such as '.xml,.txt'; empty claims every extension
detect	a function from the file’s text to True or False, used when an extension is shared
parse	a function from a path to (pages, meta), used instead of the default reader
chunker	a function returning a chonkie chunker, used instead of the 512-byte default
mode	the structure mode for build_tree, used instead of detecting one
kind	the document kind recorded on the document’s row, so searches can filter on it
meta	a function from a chunk’s text to a dict, stored in the chunk’s metadata column

meta lets a profile attach facts that only it can work out. ganapati’s Sanskrit profile stores each verse’s metre there. The metadata column is part of the full-text index that get_store builds, so those facts become searchable and filterable with no change to the table.

profile_for

profile_for returns the first registered profile that claims a file, or None. It checks the extension first and then calls detect on up to the first 60,000 characters of the file. An extension alone is never enough to decide: .xml may be TEI, a vedicreader export or something unrelated, and .txt may be anything. A profile that has no detect and shares its extension with another can still be chosen by passing name=. register_profile adds a profile, replacing any with the same name.

---

source

file_parse

def file_parse(
    p:pathlib.Path=None, # path to the file
    *, code:str=None, # code string to parse
    imports:bool=False, # also return each `from ... import ...` statement as a chunk
    assigns:bool=False, # also return each top-level assignment as a chunk
)->fastcore.foundation.L: # passed to pyparse for `.py` files: `imports`, `assigns`

Chunks for one file, read by the reader its extension calls for; an empty list for unknown extensions.

---

source

profile_for

def profile_for(
    path:NoneType=None, # file whose extension and opening text select the profile
    text:str=None, # already-read text, when there is no file
    name:str=None, # ask for one by name, skipping detection
)->__main__.Profile | None:

The first registered profile that claims this file or text, or None for the default pipeline.

---

source

register_profile

def register_profile(
    p:__main__.Profile
)->__main__.Profile:

Add a profile to the registry, replacing any profile with the same name.

---

source

load_plugins

def load_plugins(
    eps:NoneType=None
)->fastcore.foundation.L:

Run every installed litesearch.profiles entry point once, so other packages can register profiles.

---

source

Profile

def Profile(
    name:str, exts:str='', parse:<built-in function callable>=None, chunker:<built-in function callable>=None,
    mode:str=None, detect:<built-in function callable>=None, kind:str=None, meta:<built-in function callable>=None
)->None:

How to read one kind of document: its reader, structure mode, chunker and extra metadata.

Another package can add profiles without litesearch importing it, and without importing litesearch itself. It declares an entry point in the litesearch.profiles group that names a function, and that function calls register_profile. The first call to profile_for runs load_plugins, which calls each such function once. A plugin that raises is skipped with a warning, and the others still load. ganapati registers its Sanskrit profiles this way:

[project.entry-points."litesearch.profiles"]
ganapati = "ganapati.lemma:register_profiles"

from types import SimpleNamespace
from fastcore.test import test_eq
ep = SimpleNamespace(name='zz', load=lambda: lambda: register_profile(Profile('zz_test', exts='.zz')))
bad = SimpleNamespace(name='bad', load=lambda: 1/0)
import warnings
with warnings.catch_warnings(record=True) as ws:
    warnings.simplefilter('always')
    test_eq(load_plugins([ep, bad]), ['zz'])             # a broken plugin warns, the rest still load
assert any('bad' in str(w.message) for w in ws)
test_eq(profile_for('a.zz').name, 'zz_test')
PROFILES.pop('zz_test');

ipynb_parse returns every cell of a notebook, code, markdown and raw alike. Here is the first cell of this project’s core notebook:

ipynb_parse(Path('01_core.ipynb'))[0]

{'content': '---\ndescription: Building blocks for litesearch\noutput-file: core.html\ntitle: core\n\n---\n\n',
 'metadata': {'path': '01_core.ipynb',
  'uploaded_at': 1787911231.2660935,
  'lang': '.ipynb',
  'type': 'raw'}}

pyparse reads a Python file, or a string of code passed as code=, and returns one chunk per top-level function and class. Methods stay inside their class’s chunk, and a function defined inside another function stays inside it. The metadata gives the name, the node type (FunctionDef, ClassDef and so on), the first and last line, the file’s path and its modification time. With code= there is no file, so path is the string 'None' and uploaded_at is None.

It reads only the top level of the module and splits the source into lines once per file. On 400 files of the transformers package that took parsing from 23.4 s to 3.3 s with the same chunks.

txt = """
from fastcore.all import *
a=1
class SomeClass:
    def __init__(self,x): store_attr()
    def method(self): return self.x + a
 """
pyparse(code=txt)

[{'content': 'class SomeClass:\n    def __init__(self,x): store_attr()\n    def method(self): return self.x + a', 'metadata': {'path': 'None', 'uploaded_at': None, 'name': 'SomeClass', 'lang': '.py', 'type': 'ClassDef', 'lineno': 4, 'end_lineno': 6}}]

With imports=True it also returns each from ... import ... statement. A plain import x statement is not included.

pyparse(code=txt, imports=True)

[{'content': 'from fastcore.all import *', 'metadata': {'path': 'None', 'uploaded_at': None, 'name': None, 'lang': '.py', 'type': 'ImportFrom', 'lineno': 2, 'end_lineno': 2}}, {'content': 'class SomeClass:\n    def __init__(self,x): store_attr()\n    def method(self): return self.x + a', 'metadata': {'path': 'None', 'uploaded_at': None, 'name': 'SomeClass', 'lang': '.py', 'type': 'ClassDef', 'lineno': 4, 'end_lineno': 6}}]

pyparse(code=txt, assigns=True)

[{'content': 'a=1', 'metadata': {'path': 'None', 'uploaded_at': None, 'name': None, 'lang': '.py', 'type': 'Assign', 'lineno': 3, 'end_lineno': 3}}, {'content': 'class SomeClass:\n    def __init__(self,x): store_attr()\n    def method(self): return self.x + a', 'metadata': {'path': 'None', 'uploaded_at': None, 'name': 'SomeClass', 'lang': '.py', 'type': 'ClassDef', 'lineno': 4, 'end_lineno': 6}}]

For a JavaScript file file_parse hands over to codesigs, which returns each function’s signature with its body replaced by {...}. The top-level const and let lines are not chunks.

def parse_js():
    from textwrap import dedent
    from tempfile import NamedTemporaryFile as ntf

    js_code = dedent("""
        const htmlElement = document.documentElement;
        const mediaQuery = window.matchMedia('(prefers-color-scheme: dark)');
        const vr = '__VR__';
        let __VR__ = JSON.parse(localStorage.getItem(vr) || '{{__state__}}');
        function storeState(key, value) {__VR__[key] = value; localStorage.setItem(vr, JSON.stringify(__VR__));}
        function getState(key){return __VR__[key];}
        function removeState(key) {delete  __VR__[key]; localStorage.setItem(key, JSON.stringify(__VR__)); }
        function setTheme(color,fn=null, ...args) {
            if (color === null || color === undefined) {return;}
            htmlElement.classList.remove(__VR__.theme);
            htmlElement.classList.add(color);
            storeState('theme', color);
            if (typeof fn === 'function') {fn(...args);}
        }
    """)

    with ntf('w+', suffix='.js', encoding='utf-8') as f:
        f.write(js_code)
        f.flush()  # Ensure content is readable by file_parse from disk
        print(file_parse(Path(f.name)))

parse_js()

[{'content': 'function storeState(key, value) {...}', 'metadata': {'path': '/tmp/tmpfpafitq6.js', 'uploaded_at': 1792653836.8016536, 'lang': '.js'}}, {'content': 'function getState(key) {...}', 'metadata': {'path': '/tmp/tmpfpafitq6.js', 'uploaded_at': 1792653836.8016536, 'lang': '.js'}}, {'content': 'function removeState(key) {...}', 'metadata': {'path': '/tmp/tmpfpafitq6.js', 'uploaded_at': 1792653836.8016536, 'lang': '.js'}}, {'content': 'function setTheme(color,fn=null, ...args) {...}', 'metadata': {'path': '/tmp/tmpfpafitq6.js', 'uploaded_at': 1792653836.8016536, 'lang': '.js'}}]

globsafe takes the parameters of fastcore’s globtastic and returns the same files, but skips an entry it cannot read. globtastic raises OSError at a symlink that points at itself, and returns a dangling link as if it were a file. globsafe leaves out both, and an unreadable folder, and does not enter a symlinked folder that leads back into one of its own ancestors. pkg2files and dir2files use it.

---

source

globsafe

def globsafe(
    path:pathlib.Path | str='.', # folder to walk
    recursive:bool=True, maxdepth:int=None, symlinks:bool=True, file_glob:str=None, file_re:str=None,
    path_glob:str=None, path_re:str=None, folder_re:str=None, skip_file_glob:str=None, skip_file_re:str=None,
    skip_folder_re:str=None, func:<built-in function callable>=walk_join, ret_folders:bool=False, sort:bool=True,
    types:str | list=None, exts:str | list=None
)->fastcore.foundation.L:

globtastic that skips symlink loops, dangling links and unreadable folders instead of raising.

---

source

pkg2files

def pkg2files(
    pkg:str, # installed package, by import or distribution name
    skip_file_re:str='^[.]|^(?:setup\\.py|conftest\\.py)$', # regex of file names to skip
    skip_folder_re:str='^[.]|^(?:tests?|examples?|docs?|build|dist|_proc|_docs|_site|_site_libs|node_modules|__pycache__)$', # regex of folder names to skip
    func:type=Path, # applied to each path, such as Path or str
    path:list=None, # site-packages folders to look in; None looks in the running interpreter
    *, recursive:bool=True, maxdepth:int=None, symlinks:bool=True, file_glob:str=None, file_re:str=None,
    path_glob:str=None, path_re:str=None, folder_re:str=None, skip_file_glob:str=None, ret_folders:bool=False,
    sort:bool=True, types:str | list=None, exts:str | list=None
)->fastcore.foundation.L: # passed to globsafe, such as `exts=code_exts`

Every file in an installed package’s folder, skipping tests, docs, build output and hidden files.

pkg2files('fastlite', exts=code_exts)

[Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/fastlite/__init__.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/fastlite/_modidx.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/fastlite/core.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/fastlite/kw.py')]

pkg2files('protobuf')

[Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/_upb/_message.abi3.so'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/__init__.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/any.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/any_pb2.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/api_pb2.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/compiler/__init__.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/compiler/plugin_pb2.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/descriptor.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/descriptor_database.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/descriptor_pb2.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/descriptor_pool.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/duration.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/duration_pb2.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/empty_pb2.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/field_mask_pb2.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/internal/__init__.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/internal/api_implementation.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/internal/builder.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/internal/containers.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/internal/decoder.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/internal/encoder.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/internal/enum_type_wrapper.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/internal/extension_dict.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/internal/field_mask.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/internal/message_benchmark.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/internal/message_listener.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/internal/python_edition_defaults.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/internal/python_message.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/internal/testing_refleaks.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/internal/type_checkers.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/internal/well_known_types.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/internal/wire_format.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/json_enumvalue_options_pb2.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/json_format.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/json_options_pb2.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/message.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/message_factory.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/proto.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/proto_builder.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/proto_json.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/proto_text.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/pyext/__init__.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/pyext/cpp_message.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/reflection.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/runtime_version.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/service_reflection.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/source_context_pb2.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/struct_pb2.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/symbol_database.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/testdata/__init__.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/text_encoding.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/text_format.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/timestamp.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/timestamp_pb2.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/type_pb2.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/unknown_fields.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/util/__init__.py'), Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/google/protobuf/wrappers_pb2.py')]

---

source

dir2files

def dir2files(
    dir:str, # folder to walk
    skip_file_re:str='^[.]|^(?:setup\\.py|conftest\\.py)$', # regex of file names to skip
    skip_folder_re:str='^[.]|^(?:tests?|examples?|docs?|build|dist|_proc|_docs|_site|_site_libs|node_modules|__pycache__)$', # regex of folder names to skip
    func:type=Path, # applied to each path, such as Path or str
    *, path:pathlib.Path | str='.', # folder to walk
    recursive:bool=True, maxdepth:int=None, symlinks:bool=True, file_glob:str=None, file_re:str=None,
    path_glob:str=None, path_re:str=None, folder_re:str=None, skip_file_glob:str=None, ret_folders:bool=False,
    sort:bool=True, types:str | list=None, exts:str | list=None
)->fastcore.foundation.L: # passed to globsafe, such as `exts=` or `skip_file_glob=`

Every file under a folder, skipping tests, docs, build output and hidden files.

dir2files(repo_root(), exts=code_exts.split(','), skip_file_glob='*.ipynb')

[Path('/Users/71293/code/litesearch/evals/__init__.py'), Path('/Users/71293/code/litesearch/evals/boldhead.py'), Path('/Users/71293/code/litesearch/evals/build.py'), Path('/Users/71293/code/litesearch/evals/cluster_eval.py'), Path('/Users/71293/code/litesearch/evals/corpus.py'), Path('/Users/71293/code/litesearch/evals/decide.py'), Path('/Users/71293/code/litesearch/evals/encoders.py'), Path('/Users/71293/code/litesearch/evals/extractor_eval.py'), Path('/Users/71293/code/litesearch/evals/extractor_sig.py'), Path('/Users/71293/code/litesearch/evals/fastlate.py'), Path('/Users/71293/code/litesearch/evals/fetch_corpus.py'), Path('/Users/71293/code/litesearch/evals/ingest_bench.py'), Path('/Users/71293/code/litesearch/evals/latechunk.py'), Path('/Users/71293/code/litesearch/evals/multihop.py'), Path('/Users/71293/code/litesearch/evals/queries.py'), Path('/Users/71293/code/litesearch/evals/refindex.py'), Path('/Users/71293/code/litesearch/evals/report.py'), Path('/Users/71293/code/litesearch/evals/run.py'), Path('/Users/71293/code/litesearch/evals/score.py'), Path('/Users/71293/code/litesearch/evals/tables.py'), Path('/Users/71293/code/litesearch/litesearch/__init__.py'), Path('/Users/71293/code/litesearch/litesearch/_modidx.py'), Path('/Users/71293/code/litesearch/litesearch/api.py'), Path('/Users/71293/code/litesearch/litesearch/cli.py'), Path('/Users/71293/code/litesearch/litesearch/core.py'), Path('/Users/71293/code/litesearch/litesearch/data.py'), Path('/Users/71293/code/litesearch/litesearch/postfix.py'), Path('/Users/71293/code/litesearch/litesearch/quality.py'), Path('/Users/71293/code/litesearch/litesearch/sanskrit.py'), Path('/Users/71293/code/litesearch/litesearch/topics.py'), Path('/Users/71293/code/litesearch/litesearch/tree.py'), Path('/Users/71293/code/litesearch/litesearch/utils.py')]

A folder with a file symlinked to itself, a dangling link and a link back to its parent still lists its real files. On a tree without such entries globsafe returns what globtastic does.

_lp = Path(tempfile.mkdtemp())/'loops'; (_lp/'sub').mkdir(parents=True)
for _n in ('a.py', 'sub/b.py'): (_lp/_n).write_text('x = 1\n')
(_lp/'loop.jpg').symlink_to(_lp/'loop.jpg'); (_lp/'gone.py').symlink_to(_lp/'nowhere.py'); (_lp/'sub'/'up').symlink_to(_lp)
_fc = Path(spec('fastcore').origin).parent
test_eq(globsafe(_fc, exts='py', skip_folder_re='__pycache__'), globtastic(_fc, exts='py', skip_folder_re='__pycache__'))
dir2files(_lp).map(lambda p: str(p.relative_to(_lp)))

With path= a package is looked up in those site-packages folders instead of the running interpreter’s, and its version comes from the dist-info found there. kkfake is installed in neither.

_sp = Path(tempfile.mkdtemp())/'site-packages'; (_sp/'kkfake').mkdir(parents=True)
(_sp/'kkfake'/'__init__.py').write_text('def hello(): return 1\n')
(_sp/'kkfake-1.0.dist-info').mkdir()
(_sp/'kkfake-1.0.dist-info'/'METADATA').write_text('Metadata-Version: 2.1\nName: kkfake\nVersion: 1.0\nAuthor-email: a@b.co\n')
test_eq(spec('kkfake'), None)
test_eq(pkg_dist('kkfake', [_sp]).version, '1.0')
pkg2files('kkfake', path=[_sp]).map(lambda p: p.name)

---

source

dir2chunks

def dir2chunks(
    dir:str, # directory path
    imports:bool=False, # include import statements as code chunks
    assigns:bool=False, # include top-level assignments as code chunks
    n_workers:int=None, # parse workers; 0 is serial, None picks by file count
    threadpool:bool=False, # threads rather than processes (see `_parse_files`)
    *, skip_file_re:str='^[.]|^(?:setup\\.py|conftest\\.py)$', # regex of file names to skip
    skip_folder_re:str='^[.]|^(?:tests?|examples?|docs?|build|dist|_proc|_docs|_site|_site_libs|node_modules|__pycache__)$', # regex of folder names to skip
    func:type=Path, # applied to each path, such as Path or str
    path:pathlib.Path | str='.', # folder to walk
    recursive:bool=True, maxdepth:int=None, symlinks:bool=True, file_glob:str=None, file_re:str=None,
    path_glob:str=None, path_re:str=None, folder_re:str=None, skip_file_glob:str=None, ret_folders:bool=False,
    sort:bool=True, types:str | list=None, exts:str | list=None
)->fastcore.foundation.L: # passed to dir2files, such as `exts=` or `types=`

Chunks for every file under a folder, with the folder in their metadata.

---

source

pkg2chunks

def pkg2chunks(
    pkg:str, # package name
    imports:bool=False, # include import statements as code chunks
    n_workers:int=None, # parse workers; 0 is serial, None picks by file count
    threadpool:bool=False, # threads rather than processes (see `_parse_files`)
    path:list=None, # site-packages folders to read the package from; None uses the running interpreter
    **kw
)->fastcore.foundation.L: # passed to pkg2files, such as `exts=code_exts`

Chunks for every file of an installed package, with the package name and version in their metadata.

pkg2chunks reads a whole installed package, which is how you make a library’s source searchable. dir2chunks does the same for a folder, such as your own repository. Both find the files with pkg2files or dir2files, read each with file_parse, and add to the metadata: the package name and version, or the folder.

The two file listers skip hidden files and folders, setup.py, conftest.py, and folders named test, tests, example, examples, doc, docs, build, dist, node_modules, __pycache__, and nbdev’s generated _proc, _docs, _site and _site_libs. They return every other file, so pass exts=code_exts (or a list of extensions) to keep to code.

From 64 files up, the files are read in a pool of processes, one per CPU. Processes are used rather than threads because parsing holds Python’s global interpreter lock the whole time: over 2,170 files, four threads ran at 0.76 times the speed of one, and four processes at 3.37 times. Pass n_workers=0 to read in the current process.

pkg2chunks('fastlite').filter(lambda d: d['metadata']['type']=='FunctionDef')[0]

{'content': 'def t(self:Database): return _TablesGetter(self)',
 'metadata': {'path': '/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/fastlite/core.py',
  'uploaded_at': 1773452878.5692947,
  'name': 't',
  'lang': '.py',
  'type': 'FunctionDef',
  'lineno': 44,
  'end_lineno': 44,
  'package': 'fastlite',
  'version': '0.2.4'}}

pkg2files('pdf_oxide')[0]

Path('/Users/71293/code/litesearch/.venv/lib/python3.13/site-packages/pdf_oxide/__init__.py')

dir2chunks('../nbs', types='ipynb')[0]

{'content': '---\ndescription: Fixes for post import issues\noutput-file: postfix.html\ntitle: postfix\n\n---\n\n',
 'metadata': {'path': '../nbs/00_postfix.ipynb',
  'uploaded_at': 1773099020.1621501,
  'lang': '.ipynb',
  'type': 'raw',
  'dir': '../nbs'}}

---

source

installed_packages

def installed_packages(
    nms:list=None, # package names or requirement strings; None lists every installed package
    pyproject:bool=False, # also include the dependencies in the current repository's pyproject.toml
    xtras:str | None=None, # comma-separated `[dependency-groups]` from pyproject.toml to include too
    path:list=None, # site-packages folders to look in; None looks in the running interpreter
)->fastcore.foundation.L:

The names of installed packages, all of them or only those asked for, leaving out the standard library.

installed_packages tells you which packages in an environment you could index with pkg2chunks. Called with a list of names, it returns those that are installed and drops the rest. With pyproject=True it adds the dependencies listed in the current repository’s pyproject.toml, and xtras adds named dependency groups from the same file. With no arguments it lists every installed distribution.

The standard library is left out by checking each distribution’s Author-email metadata, so a package that does not set that field is left out too. In the first example below, fstlite is a misspelling that is not installed, so only the pyproject.toml dependencies come back.

installed_packages(['fstlite'], pyproject=True, xtras='foo') # non existent package

['pdflite', 'chonkie', 'pillow', 'model2vec', 'tokenizers', 'pdf_oxide', 'fastlite', 'usearch', 'codesigs']

installed_packages(['fastlite']) # existing package

['fastlite']

installed_packages('rishi[all]')

['rishi']

path= narrows installed_packages the same way: the kkfake package from the pkg2files example above is the only one listed.

test_eq(installed_packages(['kkfake'], path=[_sp]), ['kkfake'])
installed_packages(path=[_sp])

_ls = spec('litesearch')
assert _ls and spec('litesearch[sanskrit]')
assert spec('litesearch[sanskrit]').name == _ls.name
assert 'litesearch' in installed_packages(['litesearch[sanskrit]'])

_rishi = spec('rishi')
if _rishi:
    assert spec('rishi[all]')
    assert spec('rishi[all]').name == _rishi.name
    assert 'rishi' in installed_packages(['rishi[all]'])

installed_packages()[:10] # all installed packages that are not stdlib

['shellingham', 'nbdev', 'python-fastllm', 'tornado', 'selectolax', 'threadpoolctl', 'apswutils', 'uri-template', 'rfc3339-validator', 'pexpect']

installed_packages(pyproject=True) # all installed packages that are dependencies in pyproject.toml

['chonkie', 'codesigs', 'fastlite', 'model2vec', 'tokenizers', 'pdf_oxide', 'pdflite', 'pillow', 'usearch']

installed_packages(pyproject=True, xtras='dev')

['chonkie', 'codesigs', 'fastlite', 'model2vec', 'tokenizers', 'pdf_oxide', 'pdflite', 'pillow', 'usearch', 'nbdev', 'jupyterlab', 'jupyterlab-quarto', 'notebook', 'FlashRank', 'onnx', 'onnxruntime', 'model2vec', 'datasets', 'rishi', 'koshas', 'fastprogress', 'selectolax', 'pyperclip']

Preparing a query for full-text search

SQLite’s full-text search engine, FTS5, finds rows that contain the words of a query and ranks them with bm25, a score that rewards rare words and words that appear often in a short row. Its query language gives several characters a special meaning: * marks a prefix, " quotes a phrase, parentheses group terms, ^ anchors a term to the start of a column, - belongs to the syntax for choosing columns, and OR, AND and NOT combine terms. A word may contain only letters, digits and _ unless it is quoted. A question typed by a user can therefore fail with a syntax error, and even a valid one asks for too much: by default FTS5 returns only rows that contain every word, including words such as “how” and “does”.

These functions turn a user’s text into a query FTS5 accepts and that finds more:

• clean replaces the operator characters * , " ( ) ^, and a - directly before a word, with spaces. It returns None for an empty query. Underscores are kept, so fts_search stays one word.
• kw splits the text into words with the Unicode word-boundary rules (UAX #29, through apsw), lowercases them, and drops one-letter words and the stopwords in QSTOP, a list of common English words such as “the”, “how” and “does”. If every word is a stopword, it keeps them all.
• add_wc adds * to each word so it also matches longer words that start with it: sample* matches “samples”. A word with characters other than ASCII letters, digits and _, such as pathfinder's or any Devanagari word, is quoted first, because FTS5 accepts it only as a quoted string.
• mk_wider joins the words with OR, so a row needs only one of them to match. bm25 still ranks rows that match more of the words higher.
• pre runs all four in that order: clean, kw, add_wc, mk_wider. Each step after clean can be turned off.

Database.search passes the keyword half of every search through pre unless you set fts_pre=False. evals/ measured it at +0.016 to +0.093 weighted MRR. To build a query from a list of terms you already have, rather than from a user’s sentence, use fts_or, fts_and or fts_phrases in litesearch.core, which quote each term.

---

source

pre

def pre(
    q:str, # the user's query
    wc:bool=True, # add `*` to each word for a prefix match
    wide:bool=True, # join the words with OR
    extract_kw:bool=True, # drop stopwords with `kw`
    pattern:str='[*,"\\(\\)\\^]|-(?=\\S)', # characters `clean` replaces with a space
):

Turn a user’s query into an FTS5 query: cleaned, stopwords dropped, each word a prefix, joined with OR; None when no word is left.

---

source

kw

def kw(
    q:str, # the user's query
    stop:set=None, # stopwords to drop (defaults to `QSTOP`)
):

The lowercased words of q, without stopwords, one-letter words or repeats.

---

source

mk_wider

def mk_wider(
    q:str, # space-separated words
):

Join the words with OR so a row needs only one of them to match.

---

source

add_wc

def add_wc(
    q:str, # space-separated words
):

Add * to each word for a prefix match, quoting any word that is not plain ASCII letters, digits or _.

---

source

clean

def clean(
    q:str, # the user's query
    pattern:str='[*,"\\(\\)\\^]|-(?=\\S)', # characters to replace with a space
):

Replace FTS5 operator characters in q with spaces; None when nothing is left.

The default output for “This is a sample query” is sample* OR query*: “this”, “is” and “a” are stopwords, and the two words left become prefixes joined by OR. Each later line turns off steps to show the others alone.

q = 'This is a sample query'
print('preprocessed q with defaults: `%s`' %pre(q))
print('keywords extracted: `%s`' %pre(q, wc=False, wide=False))
print('q with wild card: `%s`' %pre(q, extract_kw=False, wide=False, wc=True))
print('remove _ : `%s`' %pre('This_is_a_query_with_underscores with_uv', extract_kw=False))

preprocessed q with defaults: `sample* OR query*`
keywords extracted: `sample query`
q with wild card: `This* is* a* sample* query*`
remove _ : `This_is_a_query_with_underscores* OR with_uv*`

# A possessive used to make `pre` emit invalid FTS5, so `search` raised instead of searching.
# `pathfinder's*` is a syntax error; `"pathfinder's"*` is a prefix query on a string.
assert pre("The Pathfinder's transition") == '"pathfinder\'s"* OR transition*'

import apsw
_c = apsw.Connection(':memory:')
_c.execute("create virtual table t using fts5(content)")
_c.execute("insert into t values(?)", ("The Pathfinder's transition activities",))
_c.execute("insert into t values(?)", ('something else entirely',))
_q = pre("Pathfinder's activities")
assert len(list(_c.execute("select content from t where t match ?", (_q,)))) == 1

# a query of operator characters alone used to raise AttributeError
for q in ('***', '""', '---', '(^)', '   '): assert pre(q) is None, q

Building a PDF from images

Photos or scans of pages often arrive as image files. images_to_pdf puts a list of images into one PDF, one image per page, so they can go through the same PDF pipeline as any other document, OCR included. It accepts Pillow images, image bytes or file paths, and writes the PDF itself from the PNG data, needing only Pillow and the standard library. img2png converts one image to RGB PNG bytes, and png_det reads a PNG’s width, height and compressed pixel data, which a PDF can embed unchanged.

The functions were written when pdf_oxide’s Pdf.from_image methods produced an empty 1.3 KB file. With pdf_oxide 0.3.77, Pdf.from_image_bytes returns a full-size file, so either route now works.

---

source

images_to_pdf

def images_to_pdf(
    imgs, # Pillow images, image bytes or file paths
    output:NoneType=None, # path to write the PDF to; the bytes are returned either way
)->bytes:

A PDF with one image per page and no text layer.

---

source

png_det

def png_det(
    png:bytes
):

The width, height and compressed pixel data (the joined IDAT chunks) of a PNG.

---

source

img2png

def img2png(
    img
)->bytes:

An image, given as a Pillow image, bytes or a path, as RGB PNG bytes.

import tempfile, os

# find PDF relative to nbs/ (normal Jupyter) or project root (nbdev_test)
_pdf = next((p for p in ['pdfs/attention_is_all_you_need.pdf', 'nbs/pdfs/attention_is_all_you_need.pdf'] if Path(p).exists()), None)
assert _pdf, 'attention_is_all_you_need.pdf not found'

arxiv = PdfDocument(_pdf)

# pdf_texts and pdf_markdown
md_pages = arxiv.pdf_markdown()
assert len(md_pages) == arxiv.page_count(), f'Expected {arxiv.page_count()} pages, got {len(md_pages)}'
assert any('attention' in p.lower() for p in md_pages)

txt_pages = arxiv.pdf_texts()
assert len(txt_pages) == arxiv.page_count()
assert any('attention' in t.lower() for t in txt_pages)

links = arxiv.pdf_links()
assert isinstance(links, L)
print(f'pdf_texts: {len(txt_pages)} pages, pdf_markdown: {len(md_pages)} pages, pdf_links: {len(links)} links')

# images_to_pdf: build synthetic scanned PDF from a real page image
with tempfile.TemporaryDirectory() as tmp:
    arxiv.to_markdown(2, include_images=True, embed_images=False, image_output_dir=tmp)
    img_bytes = open(os.path.join(tmp, os.listdir(tmp)[0]), 'rb').read()

_scanned = Path(_pdf).parent/'scanned_test.pdf'
images_to_pdf([img_bytes], output=str(_scanned))
scanned = PdfDocument(str(_scanned))
assert scanned.extract_text(0).strip() == '', 'scanned PDF should have no native text'
assert _scanned.stat().st_size > 100_000, 'scanned PDF should contain the embedded image'
print('images_to_pdf test passed:', _scanned)

Dictionary used where Stream expected, treating as empty stream
Dictionary used where Stream expected, treating as empty stream
Dictionary used where Stream expected, treating as empty stream
Dictionary used where Stream expected, treating as empty stream

pdf_texts: 15 pages, pdf_markdown: 15 pages, pdf_links: 18 links
images_to_pdf test passed: pdfs/scanned_test.pdf

---

source

mv_skill_md

def mv_skill_md(
    dry_run:bool=True, dir:NoneType=None
)->None:

Copy the bundled SKILL.md into the .agents, .claude and .Codex skill folders, or with dry_run only print where.

import io, tempfile, contextlib
from litesearch import data as _data

# dry_run used to print "Installed" too; the module's copy is used because it finds SKILL.md beside its own file
with tempfile.TemporaryDirectory() as tmp:
    with contextlib.redirect_stdout(io.StringIO()) as o: _data.mv_skill_md(dry_run=True, dir=tmp)
    assert o.getvalue().startswith('Would copy') and 'Installed' not in o.getvalue(), o.getvalue()
    assert not list(Path(tmp).rglob('SKILL.md'))
    _data.mv_skill_md(dry_run=False, dir=tmp)
    for d in ['.agents', '.claude', '.Codex']: assert 'litesearch' in (Path(tmp)/d/'skills/litesearch/SKILL.md').read_text()