Hindi SLM v001
A 46M-parameter Hindi-only causal language model trained from scratch on the ai4bharat/sangraha Hindi corpus. Designed for edge deployment (Raspberry Pi 8 GB) while producing fluent Devanagari text.
Model Details
| Property | Value |
|---|---|
| Architecture | LlamaForCausalLM (decoder-only transformer) |
| Parameters | 46.54M (unique) / 62.93M (with untied lm_head) |
| Hidden size | 512 |
| Layers | 10 |
| Attention heads | 8 (GQA with 2 KV heads) |
| Intermediate size | 1,536 (SwiGLU) |
| Context length | 512 tokens |
| Vocabulary | 32,000 (Hindi Unigram SentencePiece) |
| Precision | bfloat16 |
| Tokenizer | vaibhavmaurya/hindi-slm-tokenizer-v001 |
Training
| Property | Value |
|---|---|
| Dataset | ai4bharat/sangraha (hin/verified), ~1 GB |
| Steps | 50,000 |
| Effective batch size | 32 (batch=8 × grad_accum=4) |
| Learning rate | 3e-4 with cosine decay, 1000-step warmup |
| Weight decay | 0.1 |
| Hardware | NVIDIA RTX 3000 Ada (8 GB VRAM), bfloat16 |
| Final train loss | 3.4323 |
Evaluation
Evaluated on held-out val and test splits (98/1/1 split of the training corpus):
| Metric | Value |
|---|---|
| Val perplexity | 38.91 |
| Test perplexity | 38.63 |
| Avg Devanagari ratio | 0.765 |
| UNK tokens | 0.0 |
A val ≈ test perplexity confirms no overfitting. PPL < 50 is rated EXCELLENT for a 46M-parameter model.
Inference
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
model_id = "vaibhavmaurya/hindi-slm-v001"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
)
prompts = [
"आज का समाचार यह है कि",
"भारत की राजधानी नई दिल्ली है।",
"नमस्ते! आप कैसे हैं?",
"एक बार की बात है,",
"चाँद की रोशनी में,",
]
for prompt in prompts:
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
output_ids = model.generate(
**inputs,
max_new_tokens=100,
temperature=0.8,
top_p=0.9,
repetition_penalty=1.1,
do_sample=True,
)
generated = tokenizer.decode(output_ids[0], skip_special_tokens=True)
print(generated)
print("---")
CPU-only inference (Raspberry Pi / laptop)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.float32, # bfloat16 not supported on CPU
device_map="cpu",
)
Streaming generation
from transformers import TextStreamer
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
inputs = tokenizer("भारत की राजधानी", return_tensors="pt").to(model.device)
model.generate(
**inputs,
max_new_tokens=150,
temperature=0.8,
top_p=0.9,
repetition_penalty=1.1,
do_sample=True,
streamer=streamer,
)
Intended Use
- Hindi text completion and generation
- Educational and research use for small Hindi LMs
- Edge deployment experiments (Raspberry Pi, microcontrollers with sufficient RAM)
- Foundation for Hindi instruction-tuning
Limitations
- Context window is 512 tokens — unsuitable for long-document tasks
- Trained on web/Wikipedia-style Hindi; may not handle dialects, transliterated text, or code-switching well
- No RLHF or instruction tuning — outputs are continuations, not instruction-following responses
- Not evaluated for factual accuracy; may hallucinate
License
Apache 2.0