Telugu-vaachakam
A Telugu-only BERT masked-language model trained from scratch for Telugu language understanding and fine-tuning.
Model summary
- Architecture: BERT masked language model
- Parameters: approximately 268M
- Vocabulary: 64,000 Telugu WordPiece tokens
- Maximum sequence length: 512
- Intended use: masked-token prediction, embeddings, NER, classification, and extractive QA fine-tuning
Measured benchmark
On the google/xtreme PAN-X.te Telugu NER test split, using the same fine-tuning protocol for both models:
| Model | Entity F1 | Precision | Recall | Token accuracy |
|---|---|---|---|---|
| Telugu-vaachakam | 69.71% | 66.06% | 73.78% | 92.32% |
| Public Telugu BERT baseline | 57.42% | 54.67% | 60.46% | 88.09% |
The classification head was fine-tuned on the PAN-X.te training split and evaluated on its separate test split. These results are a benchmark comparison, not a zero-shot evaluation. The baseline is described generically here to keep the model card focused on Telugu-vaachakam rather than another project's identity.
MLM diagnostic
An in-domain masked-language-model diagnostic measured loss 2.5262, perplexity 12.51, and masked-token accuracy 54.30%. This is not a leakage-free held-out score because the pretraining run did not reserve a validation split.
No claims are made here about sentiment, topic classification, QA, or overall state-of-the-art performance because those tasks were not evaluated.
Usage
from transformers import AutoTokenizer, AutoModelForMaskedLM
tokenizer = AutoTokenizer.from_pretrained("veeranool/Telugu-vaachakam")
model = AutoModelForMaskedLM.from_pretrained("veeranool/Telugu-vaachakam")
text = "తెలుగు [MASK] ద్రావిడ భాష."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)
Limitations
This is an encoder-only BERT model, not a text-generation or chat model. Performance outside Telugu and on tasks not listed above has not been established. The model may reflect biases or noise in its training data.