mmbert-tsawa-v6-nofeat
Token classifier that finds Tsawa (རྩ་བ་, root text) spans in Tibetan Buddhist
commentaries. Fine-tuned from jhu-clsp/mmBERT-base. One model per layer: this model detects
Tsawa only.
A commentary lifts a verse or a piece of prose out of the root text, quotes it, and then explains it. That quoted piece is the Tsawa. Quotations from other works look almost the same but are not Tsawa, and telling the two apart is the hard part.
Labels
O = 0, B-TSAWA = 1, I-TSAWA = 2 (BIO).
Training data
Yontenn/formatting-tsawa-v6
(revision 2fe4a63e, the last commit that changed the data files):
8192-token windows, stride 5120, mmBERT tokenizer. 84 / 18 / 22 books for train / validation /
test (13,129 / 1,700 / 2,734 Tsawa spans). Spans were cleaned before tokenizing: 8,513 spans had an edge snapped
to syllable boundaries, neighbouring spans separated only by punctuation merged (806 merged spans),
88 books with fewer than 10 spans dropped, and 436 quotation-like spans masked out of the loss.
The split is book-level with books that share a title or root text kept together. See the dataset
card.
Training
Up to 15 epochs with early stopping (patience 3 epochs, evaluated 4 times per epoch), learning rate 1e-5, batch size 8, weight decay 0.01, gradient clip 0.3, warmup 6%, bf16, seed 42, class-weighted token cross-entropy with inverse-frequency weights (O 1.0, B 1190.7, I 13.5). The best checkpoint (epoch 6.9) was chosen on validation IoU 0.5 F1. "nofeat" means it was trained without the extra hand-made features column that the dataset also carries; that side experiment did not help.
Results
Metric: span F1 at IoU 0.5 in character offsets, greedy one-to-one matching, Viterbi decoding with
a break penalty of 4.0. Test scores are whole-book: each book is decoded once and duplicates from
overlapping windows are removed. Machine-readable test metrics, with the exact dataset revision, are in
training/test_metrics.json.
| test split | F1 | precision | recall |
|---|---|---|---|
| all 21 books (1,987 gold spans) | 0.596 | 0.485 | 0.774 |
| old batch (10 books) | 0.573 | 0.476 | 0.720 |
| new batch (11 books) | 0.629 | 0.497 | 0.856 |
| median book | 0.718 | ||
| all 22 books, with IF3ACC3E1 (2,734 gold spans) | 0.521 | 0.485 | 0.562 |
One test book, IF3ACC3E1, is an interlinear commentary that is annotated a word at a time (a median of 2 syllables, against 29 in the other test books). That is not the passage-level task this model is built for, so the headline is the score on the other 21 books, and the 22-book score is shown beside it. It has 747 gold spans and no system matches any of them, so it scores 0.000, and that is what the 22-book row includes.
The best validation score during training was F1 0.636. It was measured per window, where a span in two overlapping windows counts twice, so it is not the same protocol as the test numbers.
Caveats:
- The model over-predicts: 3,169 spans for the 1,987 gold spans in the 21 books, so precision is the weaker side. It predicts nothing in the excluded book.
Usage
The repo includes the tokenizer files, copied unchanged from jhu-clsp/mmBERT-base (revision
c5955035), so it loads on its own.
from transformers import AutoModelForTokenClassification, AutoTokenizer
tok = AutoTokenizer.from_pretrained("Yontenn/mmbert-tsawa-v6-nofeat")
model = AutoModelForTokenClassification.from_pretrained("Yontenn/mmbert-tsawa-v6-nofeat")
Decode with Viterbi rather than argmax: only legal BIO transitions (I may not follow O), and a
penalty for every span exit, which stops one span being split into pieces. A break penalty of 4.0
was used for the test scores. Long texts need 8192-token windows with stride 5120. Reference code
(src/mmbert_predict.py, src/eval_viterbi_iou.py) is in the tsawa-layer-detection repository.
Acknowledgements
Source texts were digitized and made available by the Buddhist Digital Resource Center (BDRC). We gratefully acknowledge BDRC. Annotations were prepared through OpenPecha with support from the Tsadra Foundation.