sanskrit

an FTS5 tokenizer that folds Sanskrit, so a query matches whatever script it is typed in

Why every store gets this

Before the fold, a Sanskrit query found nothing:

query	against	hits
`देवी`	`दे॒वी`, the same word accented	0
`श्रीमाता`	`śrīmātā`, the same word in IAST	0
`kuṇḍa`	`cidagnikuṇḍasambhūtā`	0

The fix is to fold both scripts to one lossy ASCII key at index time, rather than transliterate at query time. simplify casefold already turns IAST śrīmātā into srimata; deva2ascii is chosen so Devanagari श्रीमाता lands on srimata too.

This is on for every store, not only Sanskrit ones, because it costs nothing on English text and it is the largest measured retrieval win in the repository. Everything else about Sanskrit, metre, verse chunking, citations and lemmas, is in ganapati.

source

fold_token

def fold_token(
    s:str
)->str:

The ASCII index key for one token, whatever script it arrived in.

Cached because it is called once per token per FTS write and per query, and a corpus is a small vocabulary repeated: 70,413 tokens of the eval corpus are 4,153 distinct strings, so 94% of the calls are asking a question that has already been answered. It is three unicode normalisations and a per-character category scan deep, which is why it was a third of the cost of indexing a corpus with no Sanskrit in it at all. Pure function of s, so the cache is only ever a memo — same output, 5.5x fewer of them computed.

source

detect_script

def detect_script(
    s:str
)->str:

'deva', 'latn' or 'other' — enough to pick a folding path.

source

deva2ascii

def deva2ascii(
    s:str
)->str:

Devanagari to a bare ASCII key, applying the implicit a of an unmarked consonant.

source

strip_vedic

def strip_vedic(
    s:str
)->str:

Drop Vedic tone marks from Devanagari, leaving Latin diacritics alone. NFC out.

The tokenizer

The fold could have been a second indexed column, the way an application usually does it. As an FTS5 tokenizer instead it needs no schema change, no second column and no query rewriting, and it works for any store that already exists.

sanskrit_tokenizer wraps another tokenizer rather than replacing it. It emits the fold as a colocated token inside porter, so English keeps its stemming and identifiers stay whole.

source

register_sanskrit

def register_sanskrit(
    db
):

Register the sanskrit FTS5 tokenizer on a connection. Idempotent.

source

sanskrit_tokenizer

def sanskrit_tokenizer(
    con, args
):

A wrapping* FTS5 tokenizer that folds Sanskrit so a query matches regardless of script.*

Citations

CITE_RE matches a GRETIL citation, // Mn_1.1 // or || BrhUp_1,1.2 ||. It lives here because tree.py builds the document tree out of it: the citation is the verse’s address, so a Sanskrit source gets a real tree with no markup at all.

source

cite_parts

def cite_parts(
    c:str
)->tuple:

'BrhUp_1,1.2' -> ('BrhUp', ['1','1','2']) — the siglum and its hierarchy.

from litesearch.sanskrit import cite_parts

assert cite_parts('BrhUp_1,1.2') == ('BrhUp', ['1','1','2'])
assert cite_parts('Mn_1.1') == ('Mn', ['1','1'])
assert cite_parts('IsUp_4[57M]') == ('IsUp', ['4'])      # the alternate-edition number is dropped
assert cite_parts('not a citation') == ('not a citation', [])