Cognita-Mini-167M
A 167M-parameter decoder-only language model for English, Odia and Dogri, pre-trained from random initialisation on 2.42 billion tokens. The architecture follows the Qwen3 layout; no pre-trained weights were used, and the tokenizer was trained for this project.
Code, figures and the full write-up: https://github.com/ProtonPratt/Cognita-Mini-SLM
| Parameters | 167,469,952 |
| Layers × width | 12 × 896 |
| Attention | 12 query heads, 4 key/value heads (grouped-query attention) |
| Context window | 4,096 tokens |
| Vocabulary | 32,000-piece SentencePiece BPE shared by Latin, Odia and Devanagari script |
| Training data | 2.42B tokens: English 66.2%, Odia 33.4%, Dogri 0.4% |
| Training | 2 epochs, 4.76B tokens processed, 8.6 h on 2× H100 |
| Validation | loss 3.203, perplexity 24.6 on 24M held-out tokens |
Usage
from transformers import AutoModelForCausalLM, LlamaTokenizer
repo = "ProtonPrat/Cognita-Mini-167M"
tokenizer = LlamaTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)
inputs = tokenizer("ଓଡ଼ିଶା ଭାରତର ଏକ ରାଜ୍ୟ ଯାହା", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=80, do_sample=True, temperature=0.7,
top_p=0.8, repetition_penalty=1.1, no_repeat_ngram_size=3)
print(tokenizer.decode(output[0], skip_special_tokens=True))
Requires transformers>=4.56,<5 and sentencepiece. Load the tokenizer with LlamaTokenizer.
This is a plain language model: it continues text. For question-style prompts, add the instruction adapter.
Training data
| Language | Tokens | Sources |
|---|---|---|
| English | 1,602,942,297 | Wikipedia (2023-11 dump), FineWeb (CC-MAIN-2025-26) |
| Odia | 807,950,933 | AI4Bharat IndicCorp v2 and BPCC, eleven OdiaGenAI collections |
| Dogri | 10,030,800 | CIIL Dogri corpus, a Dogri Alpaca-style instruction set |
Text was NFC-normalised, stripped of HTML, URLs and control characters, and filtered to lines of more than five words. Token streams were packed into 4,096-token sequences with no padding.
Training procedure
AdamW, peak learning rate 3e-4 with 1% warm-up and cosine decay, weight decay 0.1, bfloat16 autocast, 786,432 tokens per update (192 sequences), 6,052 updates. 1% of the packed sequences were held out for validation; the best checkpoint by validation loss is released.
Evaluation
| Metric | Value |
|---|---|
| Validation perplexity (in-distribution, 24M tokens) | 24.6 |
| PIQA validation, two-way likelihood scoring (chance 50%) | 57.2% |
Limitations
- A small model trained on a small corpus: fluent, but it invents facts. No safety tuning.
- Dogri is 0.4% of the training data and is the weakest of the three languages.
- Validation text comes from the same sources as the training text, so perplexity does not measure transfer to new domains.
- Not evaluated for bias. Web and news text carry the biases of their sources.
License
CC BY-NC 4.0. Research and non-commercial use.