Klyra-64M-Midtrain
Klyra-64M-Midtrain is the continued-pretraining checkpoint of Klyra-64M.
It starts from Klyra-64M-Base and adds approximately 2B additional tokens, bringing the cumulative pretraining + midtraining exposure to about 3B tokens.
Midtraining Mixture
| Source | Approx. tokens |
|---|---|
| DCLM-Edu | 720M |
| FineWeb-Edu | 480M |
| Stack-Edu | 500M |
| FineMath | 120M |
| InfiMM-WebMath | 80M |
| Cosmopedia v2 | 100M |
Training used a 1,024-token context and BF16 precision.
Intended Use
This checkpoint is intended for:
- continued pretraining,
- downstream SFT,
- domain adaptation,
- experiments that need the strongest pre-instruction Klyra checkpoint.
It is not yet instruction tuned.
About Klyra
Klyra-64M is a compact language-model research project initiated and developed by a student of Informatics Engineering at Politeknik Negeri Jakarta (State Polytechnic of Jakarta).
The model uses a MiniMind-compatible decoder-only architecture and was trained through a staged pipeline from random initialization.
Main project: Jahirrrr/Klyra-64M
Architecture
| Component | Value |
|---|---|
| Unique runtime parameters | ~63.9M |
| Transformer layers | 8 |
| Hidden size | 768 |
| Attention heads | 8 |
| KV heads | 4 |
| Vocabulary | ~6.4K |
| Context length | 1,024 |
| MoE | No |
Parameter note: Klyra uses tied input/output embeddings at runtime. Some exported
safetensorsfiles may contain bothmodel.embed_tokens.weightandlm_head.weightas separate serialized tensors. The intended runtime model has 63,912,192 unique parameters after weight tying.