Upanishad GPT Small
A small decoder-only Transformer language model trained from scratch on a corpus of Sanskrit Upanishadic texts.
Architecture
- Decoder-only Transformer
- 4 Transformer layers
- 4 attention heads
- Hidden dimension: 128
- Context length: 128 tokens
- SentencePiece tokenizer
- Vocabulary size: 1024
Training
The model was implemented in PyTorch from scratch.
The training objective is causal language modeling:
Given:
x1 x2 x3 ... xn
the model learns to predict:
x2 x3 x4 ... x(n+1)
Important limitation
This is an experimental small language model trained on a very small corpus. It should not be interpreted as a general Sanskrit language model or as a source of authoritative interpretation of Upanishadic texts.
The model may memorize portions of the training corpus and may generate grammatically or semantically incorrect Sanskrit.
Corpus
The training corpus consists of Sanskrit Upanishadic texts obtained from GRETIL source files.
The corpus preparation and source-specific usage terms should be checked against the original sources before redistribution.
Model status
Experimental / educational.
The primary purpose of this model is to study:
- tokenization
- causal self-attention
- Transformer architecture
- Sanskrit language modeling
- overfitting and generalization
- small-scale language-model training