qwen35-mitra-dictionary-annotation

The word aligner behind MITRA-dict, a family of attested Sanskrit to English, Tibetan and Chinese dictionaries.

Given a Sanskrit sentence, segmented and lemmatized with ByT5-Sanskrit, and its translation, the model copies the translation and wraps the span that renders each Sanskrit word in that word's numbered tag. Words without a clear equivalent are left untagged.

Prompt

Insert the numbered tags into the target line, marking the word(s) that translate each numbered source word. Copy the target verbatim and omit numbers without a clear equivalent.

SANSKRIT: <1>evam</1> <2>ca</2> <3>tat</3> <4>karaṇīyam</4>
LEMMAS: <1>evam</1> <2>ca</2> <3>tad</3> <4>kṛ</4>
TIBETAN: དེ་ནི་དེ་ལྟ་བུར་བྱ་སྟེ།
OUTPUT:

Output:

དེ་ནི་<1>དེ་ལྟ་བུར</1>་<4>བྱ</4>་སྟེ།

The target line is TIBETAN:, ENGLISH: or CHINESE: (followed by a CHINESE_SEGMENTED: line with the word-segmented Chinese). Tibetan is given in Unicode. Use greedy decoding and stop at the first newline.

Training

Full fine-tuning of MITRA Qwen3.5-9B (continued pretraining on Sanskrit, Tibetan, Buddhist Chinese and Pāli) for three epochs on 9,366 sentence pairs labelled by a commercial LLM. Directions are Sanskrit to Tibetan, Chinese and English, plus Chinese to Tibetan and English to Tibetan.

Evaluation

Link-level F1 against 1,489 held-out sentence pairs labelled by the same teacher: Sanskrit to Tibetan 88.9, Sanskrit to Chinese 89.2, Sanskrit to English 86.9. These figures measure agreement with the teacher, not correctness.

Format

The weights use the Qwen3_5ForConditionalGeneration layout, which vLLM requires. The vision tower is unused.

Links

Data and evaluation protocol: https://github.com/dharmamitra/dharmamitra-lexicon

Interactive dictionary: https://lexicon.dharmamitra.org