sanskrit

an FTS5 tokenizer that folds Sanskrit, so a query matches whatever script it is typed in

The problem

Full-text search matches words exactly as they were indexed. SQLite’s FTS5, the full-text engine litesearch uses, splits text into tokens with a tokenizer and stores every token in an index; a query matches a row only when the query’s tokens are in that index. Sanskrit breaks this in three ways. The same word can be printed in Devanagari or in IAST, the Latin transliteration with diacritics. Vedic texts add tone marks above and below the letters. And many readers type a plain-English spelling with no diacritics at all. Before this module, these searches found nothing:

query	against	hits
देवी	दे॒वी, the same word with Vedic accents	0
श्रीमाता	śrīmātā, the same word in IAST	0
kuṇḍa	cidagnikuṇḍasambhūtā	0

The fix is to reduce every word, whatever its script, to a lossy ASCII key when it is indexed, and to do the same to the query. श्रीमाता, śrīmātā and srimata all become srimata, so each of them finds the other two. Nothing is transliterated at query time and no second column is added.

Every store litesearch creates uses this, Sanskrit or not. English text keeps its own tokens, so an English search behaves as before, but the extra keys do take space: on this repository’s notebooks the FTS index grows 14.7%. Metre, verse chunking, citations beyond the pattern below, and lemmas live in ganapati.

Folding a word

Folding means mapping several spellings of a word to one key. This module folds at two strengths.

fold_token is the strict fold. It removes Vedic tone marks (strip_vedic), turns Devanagari into ASCII letter by letter (deva2ascii), and strips diacritics from Latin letters. Aspirates survive: भीष्म folds to bhisma and kṛṣṇa to krsna, and the consonants ś, ṣ and s all become s. detect_script reports whether a string is mostly Devanagari or Latin.

roman_fold is the loose fold, built on fold_token for readers who type English spellings. It drops the h after a consonant (so bh, ch, dh, kh, sh, th all lose it), shortens ee to i, oo to u and any doubled vowel to one, turns w into v, and reads ri before a consonant as the vowel ṛ. Bheeshma, Bhishma, Bhīṣma and भीष्म all become bisma; Krishna and कृष्ण become krsna; chandra and चन्द्र become candra. The ri rule also catches words that never had a ṛ, so śrīmātā folds to srmata. That is harmless in an index, because the query folds the same way.

skeleton goes further still and keeps a folded word’s first letter and its consonants, with vowels and y removed and repeated letters collapsed: saundarya and soundarya both become sndr. It is too loose for an index. correct(skeleton=True) uses it as a last resort when looking for the index term a misspelled query word was meant to be.

---

source

skeleton

def skeleton(
    k:str
)->str:

Keep a folded word’s first letter and its later consonants, with repeats collapsed, so saundarya and soundarya both give sndr.

---

source

roman_fold

def roman_fold(
    s:str, doubles:bool=False, # a doubled letter single: Lakshmana and Lakṣmaṇa, Raghuvamsha and Raghuvaṃśa
    endings:bool=False, # an English plural or past ending off, leaving three letters: duties and duty
    numbers:bool=False, # a number by its value: 02 and 2
)->str:

Fold a token like fold_token, then also match plain-English spellings, so Bheeshma and भीष्म both give bisma.

---

source

fold_token

def fold_token(
    s:str
)->str:

Fold one token, in Devanagari or Latin script, to its lowercase ASCII index key.

---

source

detect_script

def detect_script(
    s:str
)->str:

Return 'deva' when Devanagari letters are at least as many as Latin ones, 'latn' when there are Latin letters, else 'other'.

---

source

deva2ascii

def deva2ascii(
    s:str
)->str:

Transliterate Devanagari to plain ASCII letters, adding the a a consonant carries when no vowel sign or virama follows it.

---

source

strip_vedic

def strip_vedic(
    s:str
)->str:

Remove Vedic tone marks from Devanagari text, leave Latin diacritics alone, and return NFC-normalised text.

assert roman_fold('02', numbers=True) == '2' and roman_fold('02') == '02'
assert skeleton(roman_fold('saundarya')) == skeleton(roman_fold('soundarya'))
assert skeleton('laharii') == 'lhr' and skeleton('sukra') != skeleton('sukta')

Reading the tests above as examples: roman_fold('Bheeshma') is bisma, the same key as भीष्म, Bhīṣma and Bhishma. Distinct names stay distinct, so agni and aṅga do not meet.

The FTS5 tokenizer below emits the roman_fold key as one more token beside each word whose strict fold is at least four letters long. That third token is what lets Bheeshma reach भीष्म and Bhīṣma in a search. On this repository’s notebooks it grows the FTS index by 14.7%, from 178,705 to 204,979 bytes.

roman_fold has three switches, all off by default and all off in the tokenizer. They are for matching a reader’s word against a list of known words, not for building an index. doubles=True collapses a doubled letter, so Raghuvamsha and Raghuvaṃśa meet. endings=True cuts an English plural or past ending when three letters remain, so duties and duty both become dut. numbers=True reads a string of digits as a number, so 02 becomes 2.

from fastcore.test import test_eq
test_eq(roman_fold('Lakshmana'), 'laksmana'); test_eq(roman_fold('Lakshmana'), roman_fold('Lakṣmaṇa'))
test_eq(roman_fold('Lakkshmana', doubles=True), roman_fold('Lakṣmaṇa'))
test_eq(roman_fold('Raghuvamsha', doubles=True), roman_fold('Raghuvaṃśa', doubles=True))
test_eq(roman_fold('duties', endings=True), roman_fold('duty', endings=True))
test_eq(roman_fold('worshipped', doubles=True, endings=True), 'vorsip')
test_eq(roman_fold('yes', endings=True), 'yes')   # three letters stay
test_eq(roman_fold('duties'), 'duties')            # both off by default
test_eq(roman_fold('11', doubles=True, numbers=True), '11')   # a digit is no letter: 11 stays eleven
test_eq(roman_fold('0100', doubles=True, numbers=True), '100')
test_eq(roman_fold('திருக்குறள்', doubles=True), roman_fold('திருக்குறள்'))   # a script fold_token leaves as it is keeps its letters

The tokenizer

An FTS5 tokenizer is the function that turns a row’s text into the tokens the index stores, and a query’s text into the tokens it looks up. FTS5 lets tokenizers wrap each other: in a chain such as porter unicode61, unicode61 splits the text into words and porter stems each word it receives, so running is stored as run.

sanskrit_tokenizer is a wrapping tokenizer of that kind. It asks the tokenizer inside it for words and passes each word on together with its fold_token key and, for longer words, its roman_fold key. FTS5 calls tokens that share a position colocated: they count as the same word, so a query matches the row through any of them. The original word is kept, so nothing an English search relied on is lost. Passing original=0 drops it and keeps only the keys.

SANSKRIT_TOKENIZE is the tokenizer’s registered name, 'sanskrit'. litesearch’s default chain, _FTS_TOKENIZE in litesearch.core, is porter simplify casefold 1 sanskrit unicodewords. Read it from the right: apsw’s unicodewords splits words by the Unicode rules and keeps fts_search whole, sanskrit adds the folded keys, simplify casefold 1 lowercases every token without removing diacritics, and porter stems English. sanskrit has to sit inside porter. If it wrapped porter instead, it would fold porter’s output, the index would hold dharmaksetre while the query dharmaksetre stems to dharmaksetr, and the search would miss. The regression test below checks exactly this case.

A store’s tokenizer is fixed when its FTS table is created. database() registers sanskrit on every connection it opens, and register_sanskrit does the same for a connection you opened yourself. A connection without it, such as Python’s sqlite3 module or the sqlite3 shell, can still read the store’s ordinary columns, but every FTS query and every insert into the store fails with error in tokenizer constructor, because the insert triggers update the FTS index.

---

source

register_sanskrit

def register_sanskrit(
    db
):

Register the sanskrit FTS5 tokenizer on a connection; calling it again does nothing.

---

source

sanskrit_tokenizer

def sanskrit_tokenizer(
    con, args
):

Build an FTS5 tokenizer that wraps another and adds each token’s folded keys beside it, so a query matches in any script.

Citations

GRETIL, the Göttingen archive of Sanskrit e-texts, marks each verse with its address between double bars or double slashes, as in // Mn_1.1 // (Manusmṛti 1.1) or || BrhUp_1,1.2 ||. CITE_RE finds these markers and cite_parts splits one into its siglum, the abbreviation for the work, and its numbered levels. litesearch.tree builds a document’s tree from them, so a GRETIL text gets chapters and verses as sections with no other markup.

---

source

cite_parts

def cite_parts(
    c:str
)->tuple:

Split a citation into its siglum and its levels: 'BrhUp_1,1.2' gives ('BrhUp', ['1','1','2']).

from litesearch.sanskrit import cite_parts

assert cite_parts('BrhUp_1,1.2') == ('BrhUp', ['1','1','2'])
assert cite_parts('Mn_1.1') == ('Mn', ['1','1'])
assert cite_parts('IsUp_4[57M]') == ('IsUp', ['4'])      # the alternate-edition number is dropped
assert cite_parts('not a citation') == ('not a citation', [])