Dhee-Pravega-OG Atom (645M)
A 645M-parameter decoder-only language model, trained from scratch (no pretrained weights of any kind) on 30B tokens across 18 languages, with a strong focus on Indic scripts. It produces fluent, coherent, correct-script text across all 12 Indic languages it was trained on, plus English, Arabic, Spanish, French, German, and Russian.
This is a base language model: strong at multilingual text generation, not yet trained for tool-calling, instructions, or chat. Use it as-is for text generation, or as a foundation to fine-tune for those downstream tasks.
Architecture
DheeForCausalLM — a decoder-only transformer assembled from three published, permissively-licensed components, combined in a way no single upstream project ships:
- Reordered sublayer norms (OLMo-2): norm applied to the output of each sublayer inside the residual, targeting activation-norm growth through depth.
- Per-head QK-norm (Qwen3): each attention head's queries/keys are normalized independently, rather than one norm shared across the whole projection — keeps attention logits stable through training.
- Local:global attention ratio (Gemma-3's published 5:1 interleave pattern): built into the architecture for efficient long-context attention.
What's original here is the combination, the tokenizer, the corpus, and the weights — the implementation derives only from Apache-2.0 reference code (transformers' OLMo-2 and Qwen3 implementations), and is verified bit-identical to stock Olmo2ForCausalLM under the degenerate configuration, confirming every inherited line behaves exactly as upstream.
| Parameters | 645,617,408 (embeddings: 163,847,680 / 25%) |
| Layers | 28 |
| Hidden size | 1280 |
| Attention heads | 10 query / 2 KV (GQA, 5:1) |
| Head dim | 128 |
| FFN size | 3456 |
| Vocab size | 128,006 (128,000 SentencePiece pieces + 6 control tokens) |
| Context length | 32,768 |
| Tied embeddings | yes |
Training
- Data: 30.006B tokens, 18 languages at ~1.667B tokens each: Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Punjabi, Sanskrit, Tamil, Telugu, English, Arabic, Spanish, French, German, Russian.
- Tokenizer: custom 128,000-piece SentencePiece model built for this project, with 6 reserved control tokens (
<|sys|>,<|user|>,<|assistant|>,<|tool_call|>,<|tool_result|>,<|end|>) for downstream chat/tool-calling formats. - Optimizer: Muon for 2-D hidden-layer matrices (196 tensors / 481.7M params), AdamW for embeddings, LM head, and norms (114 tensors / 163.9M params).
- Schedule: Warmup-Stable-Decay (WSD), peak LR 6e-4, 57,231 steps, 524,288 tokens/step, block size 8,192, gradient checkpointing, z-loss for logit-scale stability.
- Hardware: single NVIDIA RTX Pro 6000 Blackwell (24GB).
Evaluation
Held-out perplexity, computed on a clean dev split never seen during training (110 blocks per language):
| Language | Perplexity | Language | Perplexity |
|---|---|---|---|
| Hindi | 11.8 | Punjabi | 13.0 |
| Malayalam | 11.8 | Odia | 23.3 |
| Tamil | 12.8 | Assamese | 23.1 |
| Marathi | 13.9 | Gujarati | 18.2 |
| Kannada | 15.5 | Bengali | 18.6 |
| Telugu | 16.2 | Sanskrit | 36.8 |
| English | 13.7 |
Generated text stays reliably in-script across all 12 Indic languages. For best generation quality, use sampling (e.g. temperature ~0.7) rather than greedy decoding, which is prone to repetition loops on longer continuations — a common pattern at this model size, not specific to any one language.
Sanskrit trails the rest of the set on both perplexity and generation stability; treat it as the one language here that would benefit most from additional data in a future iteration.
This checkpoint has not been trained on tool-calling or dialogue data, so it is not suited for agentic or function-calling use without further fine-tuning on that kind of data — the reserved control tokens above are there for exactly that purpose.
Loading the model
Requires trust_remote_code=True (the custom architecture ships as modeling_dhee.py alongside this checkpoint):
import torch
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
"dheeyantra/dhee-pravega-og-atom",
trust_remote_code=True,
dtype=torch.bfloat16,
).to("cuda").eval()
Loading the tokenizer
Load the SentencePiece model directly and use the reserved IDs for the control tokens:
import json
import sentencepiece as spm
sp = spm.SentencePieceProcessor(model_file="tokenizer.model")
specials = json.load(open("specials.json"))["specials"]
# specials = {"<|sys|>": 128000, "<|user|>": 128001, "<|assistant|>": 128002,
# "<|tool_call|>": 128003, "<|tool_result|>": 128004, "<|end|>": 128005}
ids = sp.encode("your text here")
eos_id = specials["<|end|>"] # == 128005, matches config.json's eos_token_id
License
Apache 2.0. The architecture derives only from Apache-2.0-licensed reference code (transformers' OLMo-2 and Qwen3 implementations) with no use restrictions or derivative provisions. No pretrained weights of any kind were loaded at any point — all weights in this repository were trained from random initialization on the data described above.