# sanskrit
## Why every store gets this
Before the fold, a Sanskrit query found nothing:
query
against
hits
देवी
दे॒वी, the same word accented
0
श्रीमाता
śrīmātā, the same word in IAST
0
kuṇḍa
cidagnikuṇḍasambhūtā
0
The fix is to fold both scripts to one lossy ASCII key at index time,
rather than transliterate at query time. `simplify casefold` already
turns IAST `śrīmātā` into `srimata`;
[`deva2ascii`](https://Karthik777.github.io/litesearch/sanskrit.html#deva2ascii)
is chosen so Devanagari `श्रीमाता` lands on `srimata` too.
This is on for every store, not only Sanskrit ones, because it costs
nothing on English text and it is the largest measured retrieval win in
the repository. Everything else about Sanskrit, metre, verse chunking,
citations and lemmas, is in
[ganapati](https://github.com/vedicreader/ganapati).
------------------------------------------------------------------------
source
### fold_token
``` python
def fold_token(
s:str
)->str:
```
*The ASCII index key for one token, whatever script it arrived in.*
Cached because it is called once per token per FTS write *and* per
query, and a corpus is a small vocabulary repeated: 70,413 tokens of the
eval corpus are 4,153 distinct strings, so 94% of the calls are asking a
question that has already been answered. It is three unicode
normalisations and a per-character category scan deep, which is why it
was a third of the cost of indexing a corpus with no Sanskrit in it at
all. Pure function of `s`, so the cache is only ever a memo — same
output, 5.5x fewer of them computed.
------------------------------------------------------------------------
source
### detect_script
``` python
def detect_script(
s:str
)->str:
```
*`'deva'`, `'latn'` or `'other'` — enough to pick a folding path.*
------------------------------------------------------------------------
source
### deva2ascii
``` python
def deva2ascii(
s:str
)->str:
```
*Devanagari to a bare ASCII key, applying the implicit `a` of an
unmarked consonant.*
------------------------------------------------------------------------
source
### strip_vedic
``` python
def strip_vedic(
s:str
)->str:
```
*Drop Vedic tone marks from Devanagari, leaving Latin diacritics alone.
NFC out.*
## The tokenizer
The fold could have been a second indexed column, the way an application
usually does it. As an FTS5 tokenizer instead it needs no schema change,
no second column and no query rewriting, and it works for any store that
already exists.
[`sanskrit_tokenizer`](https://Karthik777.github.io/litesearch/sanskrit.html#sanskrit_tokenizer)
wraps another tokenizer rather than replacing it. It emits the fold as a
colocated token inside porter, so English keeps its stemming and
identifiers stay whole.
------------------------------------------------------------------------
source
### register_sanskrit
``` python
def register_sanskrit(
db
):
```
*Register the `sanskrit` FTS5 tokenizer on a connection. Idempotent.*
------------------------------------------------------------------------
source
### sanskrit_tokenizer
``` python
def sanskrit_tokenizer(
con, args
):
```
*A* wrapping\* FTS5 tokenizer that folds Sanskrit so a query matches
regardless of script.\*
## Citations
`CITE_RE` matches a GRETIL citation, `// Mn_1.1 //` or
`|| BrhUp_1,1.2 ||`. It lives here because `tree.py` builds the document
tree out of it: the citation is the verse’s address, so a Sanskrit
source gets a real tree with no markup at all.
------------------------------------------------------------------------
source
### cite_parts
``` python
def cite_parts(
c:str
)->tuple:
```
*`'BrhUp_1,1.2'` -\> `('BrhUp', ['1','1','2'])` — the siglum and its
hierarchy.*
``` python
from litesearch.sanskrit import cite_parts
assert cite_parts('BrhUp_1,1.2') == ('BrhUp', ['1','1','2'])
assert cite_parts('Mn_1.1') == ('Mn', ['1','1'])
assert cite_parts('IsUp_4[57M]') == ('IsUp', ['4']) # the alternate-edition number is dropped
assert cite_parts('not a citation') == ('not a citation', [])
```