OdiaGPT-30M (Pretrained from Scratch)
OdiaGPT-30M is an autoregressive decoder-only Transformer built and trained completely from scratch for the Odia language.
It does not use pretrained GPT-2, Llama, or Indic model weights, nor off-the-shelf tokenizers. All 31.5 million parameters began from random Gaussian initialization, and its 8,192-token BPE tokenizer was trained exclusively on a cleaned subset of the AI4Bharat Sangraha Odia corpus.
Model Summary
- Organization / Creator: CodeHima
- Architecture: Decoder-only Transformer (Llama-compatible)
- Parameters: 31,465,984 (~31.5M)
- Layers: 8
- Hidden dimension: 512 (8 attention heads, head dimension: 64)
- Feed-Forward: SwiGLU ($d_{ff} = 1536$)
- Positional Encoding: Rotary Position Embeddings (RoPE, base 10000.0)
- Normalization: RMSNorm ($\epsilon = 10^{-6}$)
- Context Length: 512
- Vocabulary: 8,192 (SentencePiece BPE with byte fallback)
- Precision: FP16 (Safetensors format)
Training Dynamics
- Dataset: AI4Bharat Sangraha (
verified/ori) — 28.5M tokens / 110M cleaned characters - Tokens Seen: ~29.5 Million tokens (1,800 steps)
- Initial Loss: 9.07 -> Final Validation Loss: 4.664 (Perplexity: 106.09)
- Hardware: Single NVIDIA Tesla T4 on Google Colab (~35,200 tokens/sec)
Usage with Hugging Face Transformers
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "CodeHima/OdiaGPT-30M"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16).cuda()
prompt = "ଓଡ଼ିଶାର ଲୋକମାନେ"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(
**inputs,
max_new_tokens=64,
temperature=0.8,
top_k=50,
top_p=0.9,
repetition_penalty=1.15,
do_sample=True
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Sample Output
Prompt:
ଆଜି ସକାଳେ
Completion: ଆଜି ସକାଳେ ସକାଳେ ଦୁଇ ଝିଅ ସହ ଆସି ଶୋଇପଡ଼ିଲା । ସେହି ସମୟରେ ବାପା ଘର ଲୋକ କୁ ଖବର ଦେଇ କହିଥିଲେ ଯେ, "ତେବେ ମାଆ, ତମେ ଆଉ କିଛି କହିଦେବୁ । ମୁଁ ଏ କଥା କିଛି ଶୁଣିନାହିଁ ।"
Limitations
- Base foundation model, not instruction-tuned.
- Developed as part of the OdiaGPT from-scratch educational curriculum.