Pothana Base 300M
A 387M parameter LLaMA-style language model trained from scratch on Telugu text.
Named after Bammera Pothana, the celebrated 15th-century Telugu poet who authored the Andhra Maha Bhagavatamu.
Developed by Dvitva AI.
Model Details
| Model | pothana-base-300M |
| Architecture | LLaMA (RoPE + SwiGLU + RMSNorm + GQA) |
| Parameters | 387M (unique) |
| Hidden size | 1024 |
| Layers | 30 unique (60 effective via weight sharing) |
| Attention heads | 16 Q / 4 KV (Grouped Query Attention) |
| Intermediate size | 2816 |
| Context length | 2048 |
| Vocab size | 48,000 |
| Tokenizer | SentencePiece Unigram (48K) |
| Training | Single GPU, bf16 mixed precision |
| Developed by | Dvitva AI |
Quick Start
Using pipeline
from transformers import pipeline
pipe = pipeline("text-generation", model="dvitvaai/pothana-base-300M", trust_remote_code=True)
result = pipe("తెలుగు భాష", max_new_tokens=50, do_sample=True, temperature=0.8)
print(result[0]["generated_text"])
Note:
trust_remote_code=Trueis required for the custom tokenizer that cleans up SentencePiece word boundary markers for readable output.
Manual loading
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model = AutoModelForCausalLM.from_pretrained("dvitvaai/pothana-base-300M")
tokenizer = AutoTokenizer.from_pretrained("dvitvaai/pothana-base-300M", trust_remote_code=True)
text = "తెలుగు భాష చాలా అందమైనది"
inputs = tokenizer(text, return_tensors="pt")
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=100,
temperature=0.8,
top_k=50,
do_sample=True,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Tokenizer
This model uses a SentencePiece Unigram tokenizer with a 48K vocabulary, trained directly on Telugu text.
- Handles raw Telugu text directly (no preprocessing needed)
- Byte-fallback for out-of-vocabulary characters
- Split digits for better number handling
- NFKC normalization
Architecture
Key features:
- Grouped Query Attention (GQA): 16 query heads, 4 KV heads — 4x KV cache reduction
- Block-wise Weight Sharing: 30 unique blocks, each used twice = 60 effective layers (MobileLLM-LS)
- SwiGLU MLP with 2816 intermediate size
- RoPE positional encoding (theta=10000.0)
- RMSNorm (no bias in any linear layer)
Training
- Data: Telugu text corpus (Sangraha dataset)
- Preprocessing: SentencePiece tokenization (raw text)
- Optimizer: AdamW (lr=3e-4, weight_decay=0.1, beta1=0.9, beta2=0.95)
- Schedule: WSD (Warmup-Stable-Decay)
- Precision: bf16 mixed precision
- Hardware: Single NVIDIA B200 GPU
Limitations
- This is a base model (not instruction-tuned) — it performs text completion, not instruction following
- Trained primarily on Telugu text; limited multilingual capability
- Small model size (387M) limits reasoning and knowledge capacity
License
Apache 2.0
Citation
If you use this model, please cite:
@misc{pothana-base-300M,
title={Pothana Base 300M: A Telugu Language Model},
author={Dvitva AI},
year={2025},
url={https://huggingface.co/dvitvaai/pothana-base-300M}
}