Avey-B Sindhi 25M
A compact Sindhi encoder pretrained from scratch with masked-language modeling.
This is an encoder-only model, not a chat model or free-text generator. It is intended for fine-tuning on Sindhi classification and token-labeling tasks, representation extraction, fill-mask experiments, and continued pretraining.
Results
| Evaluation | Result |
|---|---|
| Held-out Sindhi MLM loss | 2.9448 |
| Held-out Sindhi MLM perplexity | 19.01 |
| Masked-token top-1 / top-5 accuracy | 46.38% / 67.39% |
| WikiANN Sindhi entity F1 | 41.59% ± 2.68% |
| Batch-1, length-128 latency | 1.78 ms |
| Batch-16, length-512 throughput | 401,931 tokens/s |
| Peak VRAM at batch 16, length 512 | 148 MiB |
| Sindhi tokenizer fertility | 4.04 characters/token |
The MLM evaluation uses 15,351 deterministically selected masked tokens from the held-out HPLT validation split. WikiANN is especially small for Sindhi: only 100 train, 100 validation, and 100 test examples. NER reports mean ± sample standard deviation over 5 seeds after selecting from learning rates 2e-05, 6e-05, 0.0001, 0.0005 by mean validation F1. Each run uses 20 epochs, batch size 16, and maximum sequence length 128. Treat the NER number as a low-resource indicator rather than a broad estimate of Sindhi capability.
Sindhi WikiANN comparison
Entity-level F1 is the primary NER metric. Token accuracy is intentionally not reported for
Avey-B because the frequent non-entity (O) label can make accuracy look high even when entity
recognition is weak.
The closest published results found on the same Sindhi WikiANN target split are from Hangya et al. (EMNLP 2022). Their models were trained on English NER and transferred to Sindhi with the encoder frozen; Avey-B was instead fine-tuned end-to-end on WikiANN's 100 Sindhi training examples. The scores therefore provide useful low-resource context, but are not an apples-to-apples leaderboard.
| Model | Parameters | Sindhi supervision | Sindhi WikiANN entity F1 |
|---|---|---|---|
| Avey-B Sindhi | 24.87M | 100 target-language training examples; full fine-tuning; 5 seeds | 41.59% ± 2.68% |
| mBERT | ≈178M | Zero-shot from English; frozen encoder | 8.57% |
| eBERT | ≈186M | mBERT + 10k-token vocabulary extension and Sindhi MLM adaptation; zero-shot NER | 21.86% |
| eBERT + unsupervised alignment | ≈186M | eBERT with mined word-pair alignment; zero-shot NER | 25.71% |
The published comparison uses the same 100-example Sindhi WikiANN resource, but only Avey-B uses its target-language training split. A controlled comparison would require fine-tuning all base encoders on the identical Sindhi split, learning-rate sweep, and seeds; no such unreported scores are presented here.
Other published Sindhi NER references
The following results use different datasets and protocols and must not be numerically ranked against Avey-B's WikiANN result. They are included to make the model card more useful as a map of the small published Sindhi NER landscape.
| Model / setup | Evaluation data | Precision | Recall | Entity F1 | Token accuracy |
|---|---|---|---|---|---|
| mBERT fine-tune | mirfan899/sindhi-ner, reported evaluation set | 70.80% | 64.43% | 67.46% | 97.04% |
| mBERT, direct fine-tuning | Separate gold Sindhi NER corpus | — | — | 50% | — |
| mBERT + machine-labeled pretraining | Separate gold Sindhi NER corpus | — | — | 63% | — |
| XLM-R, direct fine-tuning | Separate gold Sindhi NER corpus | — | — | 72% | — |
| XLM-R + machine-labeled pretraining | Separate gold Sindhi NER corpus | — | — | 79% | — |
Model-size comparison
Parameter count is directly comparable even when task protocols are not. Avey-B is about 4.5× smaller than the monolingual Sindhi BERT architecture, 7.2× smaller than mBERT, and 11.2× smaller than XLM-R base.
| Base encoder | Parameters | Relative to Avey-B | Pretraining scope |
|---|---|---|---|
| Avey-B Sindhi | 24.87M | 1.0× | Sindhi HPLT 3.0 |
| Sindhi-BERT-base | ≈110.7M | 4.5× | Sindhi |
| mBERT | ≈178M | 7.2× | 104-language Wikipedia |
| XLM-R base | 278.04M | 11.2× | 100-language CommonCrawl |
Only Avey-B's latency, throughput, and VRAM were measured in this release. Runtime numbers from other hardware or software stacks are not mixed into the table.
Published MLM context
The independently released Sindhi-BERT-base reports a final perplexity of 31.66 after its S6r training session. Avey-B records 19.01 on its fixed held-out HPLT evaluation. Because the validation text, tokenizer, masking, and evaluation implementation differ, these perplexities are not directly comparable and do not establish a quality ranking.
MLM learning curve
| Checkpoint | Loss | Perplexity | Top-1 | Top-5 |
|---|---|---|---|---|
| Random initialization | 10.3985 | 32810.46 | 0.00% | 0.01% |
| Step 1,000 | 6.6934 | 807.08 | 12.13% | 21.24% |
| Step 10,000 | 3.3303 | 27.95 | 42.00% | 62.36% |
| Step 20,000 | 2.9448 | 19.01 | 46.38% | 67.39% |
Model details
- Architecture: Avey-B masked-language-model encoder
- Parameters: 24,865,804
- Checkpoint size: 94.9 MiB
- Vocabulary: 32,000-token Sindhi WordPiece tokenizer
- Training sequence length: 512 tokens
- Training steps: 20,000
- Effective tokens per optimizer step: 16,384
- Total training-token budget: 327.68M
Training data
The model was trained on the Sindhi snd_Arab portion of HPLT 3.0, using quality bins 10
and 9. The preprocessing pipeline language-filters, PII-redacts, NFC-normalizes, removes
exact duplicates, and preserves Sindhi Arabic Yeh (ي).
- Retained training documents: 32,171
- Retained validation documents: 153
- Retained characters: 251,003,680
The complete preprocessing metadata and per-shard counts are included in MANIFEST.json.
Machine-readable benchmark measurements and protocols are included in
evaluation_results.json.
Usage
from transformers import AutoModelForMaskedLM, AutoTokenizer
repo_id = "DanishMahdi/avey-b-snd"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(repo_id, trust_remote_code=True)
Limitations
The training corpus is filtered web text and can contain errors, biases, or harmful content. Evaluation coverage is limited to held-out HPLT masked-token prediction and the very small, automatically annotated Sindhi WikiANN split. Evaluate on native, domain-specific datasets before production use. HPLT licenses its dataset packaging under CC0 but does not own the underlying web documents.