Vaani — Gujarati Small Language Model (~110M)
Submission: IAIRO / PRAMANA SLM++ · Domain: Medical QA (Gujarati) · Params: 109.5M (≤1B) · Base: trained from scratch
Vaani is a 110M-parameter Gujarati language model pretrained from scratch on a Gujarati corpus, then instruction-tuned and adapted to the medical-QA domain. It is built for Gujarati instruction following and grounded (open-book) medical question answering.
1. Model & Architecture
| Field | Value |
|---|---|
| Model name / version | Vaani (med-v5, final) |
| Base model | From scratch |
| Parameter count | 109.5M |
| Layers | 12 |
| Model dim (d_model) | 768 |
| Attention | Multi-Head Attention (MHA); n_head = n_kv_head = 12, head_dim 64 |
| Normalization | RMSNorm (pre-norm) |
| Activation / FFN | SwiGLU, hidden mult 8/3 (~2048) |
| Positional encoding | RoPE (theta 10,000) |
| Context length | 2,048 |
| Embeddings | Tied input/output embeddings |
| Tokenizer | Character-level BPE, vocab 32,000, trained for Gujarati (1.30 tokens/word) |
| Precision | bf16 |
2. Domain & Task
- Domain: Medical (Gujarati).
- Primary task: Multiple-choice medical QA + Gujarati instruction following.
- End user / use case: Gujarati-language assistant for grounded medical reading comprehension (answering from provided context), not a source of unaided medical facts.
3. Data
| Field | Value |
|---|---|
| Pretraining corpus | Gujarati corpus, ~6.78B tokens (Sangraha + Wikipedia-gu + FineWeb-2; see §8) |
| Pretraining tokens seen | ~13.63B (13,000 steps × 1.05M tokens/step, ≈2 epochs) |
| Instruction SFT | sft_v4.jsonl (~150k): Wiki_Chat + Dolly + WikiHow, plus format skills (JSON / extraction / transformation / exact-count lists); see §8 |
| Medical SFT | medical_sft_v3.jsonl (~114k): MedMCQA-gu closed-book + raw-grounded + redacted-grounded + 30% general mix |
| Contamination check | domain eval uses a held-out MedMCQA-gu validation split, disjoint from SFT |
4. Training
| Field | Value |
|---|---|
| Objective | Causal LM (next-token) |
| Effective batch | 512 sequences (micro-batch 8 × grad-accum 64) × 2,048 ctx = 1.05M tokens/step |
| Steps | 13,000 (pretraining) |
| Optimizer | AdamW (betas 0.9/0.95, eps 1e-8, fused, decoupled weight decay 0.1) |
| LR schedule | WSD: warmup 700 → peak 4e-4 → cosine decay to 4e-5 (final 20%) |
| Hardware | 1× NVIDIA L4 (24 GB) |
| GPU-hours (final model path) | ~100 h (pretraining ~96 + instruction SFT v4 ~3 + medical SFT v5 ~1) |
| GPU-hours (incl. SFT ablations) | ~106 h |
5. Post-Training (SFT)
| Field | Value |
|---|---|
| SFT performed | Yes |
| Method | Full fine-tune (not LoRA) |
| Instruction SFT | 3 epochs, effective batch 32 |
| Medical SFT | 2 epochs, effective batch 32, on the instruction-tuned checkpoint |
| Chat template | વપરાશકર્તા:\n{question}\nસહાયક:\n{answer} |
6. Evaluation (self-reported internal eval)
Level 1 — Training health: PASS (all gates) — loss reduction 68.2% (10.53→3.35), convergence, no NaN/Inf, final val–train gap 2.0%.
6.1 Development progression (Vaani checkpoints)
MCQ = acc_norm, fixed n=80 Gujarati dev set. Instruction = rule-based checkers (9 buckets). Open-book = MedMCQA-gu val, closed vs with-context (raw-grounded).
| Stage | MCQ (n=80) | Instruction | Open-book (closed → w/ctx) |
|---|---|---|---|
| Instruction SFT v1 | 36.3% | 35.9% | - |
| Instruction SFT v2 | 36.3% | 46.0% | - |
| Instruction SFT v3 (targeted aug) | 31.3% | 68.7% | - |
| Format SFT v4 (JSON/extract/lists) | 28.8% | 66.7% (71.2% greedy) | - |
| Medical SFT v3 (raw-grounded) | 33.8% | 68.2% | 26.6% → 30.8% |
| Medical SFT v4 | 41.3% | 66.7% | 28.6% → 38.2% |
| Medical SFT v5 (final) | 41.3% | 69.7% | 28.8% → 44.1% |
| Medical SFT v6 (redacted) | - | - | 28.7% → 42.8% |
Instruction-following rose 36% → 70% via targeted augmentation and format skills. Open-book (grounded) accuracy rose monotonically 30.8% → 38.2% → 44.1% across medical stages, lift growing +4.2 → +9.6 → +15.3. v6 (redacted-heavy) regressed vs v5, so v5 is submitted. On the larger cross-model sets, v5 scores MCQ 38.6% (n=145) and open-book with-context 44.1% (n=1,858).
6.2 Comparison vs external models
Same harness for all. MCQ = 145-item Gujarati set (acc_norm, log-likelihood); Instruction = rule-based checkers; Open-book = MedMCQA-gu val, with vs without context (Vaani n=1,858; external models n=500).
| Model | Params | MCQ | Instruction | Open-book w/ctx | Grounding lift |
|---|---|---|---|---|---|
| Vaani (final) | 110M | 38.6% | 70% | 44.1% | +15.3 |
| gemma-2-2b-it | 2B | 37.2% | 67% | 37.8% | +9.6 |
| sarvam-1 (base) | 2B | 36.6% | 24% | 40.2% | +11.4 |
| Navarasa-2.0 | 2B | 42.8% | 54% | 35.0% | +10.0 |
| Qwen2.5-7B-Instruct | 7B | 51.0% | 53% | 44.0% | +18.0 |
Takeaways:
- Instruction-following leads the field: Vaani 70% > gemma-2B 67%, Navarasa-2B 54%, Qwen-7B 53%. Even Qwen-7B fails exact-count lists (list_3/5 0%, numbered 14%) — skills Vaani was explicitly trained on.
- Open-book comprehension is best in test: Vaani 44.1% tops Qwen-7B (44.0%) and all 2B models. A 110M model leads on grounded reading.
- General MCQ: Vaani 38.6% beats gemma-2B (37.2%) and sarvam-2B (36.6%); trails Navarasa-2B (42.8%) and Qwen-7B (51%). Closed-book recall scales with size, as expected. Vaani is built to use provided context, not recall facts unaided.
7. Reproducibility
- Weights: https://huggingface.co/pratham-commits/vaani-gujarati-slm
- Datasets: https://huggingface.co/datasets/pratham-commits/vaani-gujarati-sft-data
- Inference: eval.py (research harness) / chat.py (interactive); greedy for structured/objective tasks (temp 0), sampling for open-ended.
- Dependencies: torch 2.9.1+cu129, transformers 4.57.6, tokenizers 0.22.2, huggingface_hub<1.0 (Python 3.10)
- Known limitations / failure modes:
- Closed-book medical factual recall is limited by 110M capacity; strongest when answering from provided context (open-book/RAG).
- Structured JSON output was learned only for the fixed schemas seen in training (e.g. name/age/city) and does not generalize to arbitrary keys — not exposed as a general capability. Exact-count lists require greedy decoding; sampling with no-repeat-ngram degrades them.
- Gujarati-only; not tuned for code-switching or other Indic languages.
8. Data Sources, Licenses & Attribution
Pretraining corpus (described, not re-hosted):
- AI4Bharat Sangraha (verified / unverified / synthetic, gu) — CC BY-4.0. Khan et al., 2024 (arXiv:2403.06350).
- Wikipedia (gu, 20231101) — CC BY-SA 4.0. Wikimedia Foundation.
- FineWeb-2 (guj_Gujr) — ODC-BY. Penedo et al., 2024 (arXiv:2406.17557).
Instruction SFT sources (in sft_v4.jsonl):
- Dolly (databricks-dolly-15k, translated to Gujarati) — CC BY-SA 3.0. Databricks.
- WikiHow (Gujarati) — derived from WikiHow content (CC BY-NC-SA); see license note below.
- Wiki_Chat — Wikipedia-derived instruction/QA (CC BY-SA).
- Anudesh (AI4Bharat IndicLLMSuite) was evaluated and dropped for quality; not included.
Medical data:
- MedMCQA — Pal, Umapathi & Sankarasubbu, 2022 (arXiv:2203.14371). MedMCQA-gu is a Gujarati translation of MedMCQA items, decontaminated against the held-out validation split.
External comparison models (benchmarked only, not redistributed): google/gemma-2-2b-it (Gemma Terms of Use), sarvamai/sarvam-1, Telugu-LLM-Labs/Indic-gemma-7b-finetuned-sft-Navarasa-2.0 (Gemma Terms of Use), Qwen/Qwen2.5-7B-Instruct (Apache-2.0).
License note: Vaani's weights are released under Apache-2.0. The SFT data combines sources under different licenses (CC BY-SA, and WikiHow-derived content under a non-commercial share-alike license). See the dataset card for per-source licensing; downstream users should respect each source's terms.
9. Citations
- Khan et al. (2024), IndicLLMSuite / Sangraha, arXiv:2403.06350
- Penedo et al. (2024), FineWeb / FineWeb-2, arXiv:2406.17557
- Pal, Umapathi & Sankarasubbu (2022), MedMCQA, arXiv:2203.14371
- Hoffmann et al. (2022), Chinchilla, arXiv:2203.15556; Sardana et al. (2024), arXiv:2401.00448
- Full design-decision references: see
DECISIONS.mdin the project repo.
Acknowledgements
Built during the IAIRO / PRAMANA SLM++ bootcamp.