ArzLM-300M-Base
303,353,856-parameter decoder-only base model trained from scratch on 6,000,214,016 tokens.
Not instruction tuned. Not a chat model.
Code and loader: github.com/Entourage397/ArzLM
Do not use Transformers AutoModel
This repository is not a stock LlamaForCausalLM export. Tensors are LitGPT / ArzLM (lm_head.weight, fused attn.qkv, QK-Norm). This will not load correctly:
AutoModelForCausalLM.from_pretrained("Kymaris/ArzLM-300M-Base") # unsupported
Supported loader (from the GitHub repo). After hf download or by Hub id:
from arzlm.infer import generate, load_inference_checkpoint
loaded = load_inference_checkpoint("Kymaris/ArzLM-300M-Base", device="cuda", dtype="bf16")
print(loaded.parameters, loaded.config.block_size) # 303353856, 2048
print(generate(loaded, "The capital of France is", max_new_tokens=32)["completion"])
Or a local snapshot:
loaded = load_inference_checkpoint("./ArzLM-300M-Base", device="cuda", dtype="bf16")
Download:
hf download Kymaris/ArzLM-300M-Base --local-dir ./ArzLM-300M-Base
python -m arzlm verify-checkpoint --checkpoint ./ArzLM-300M-Base/litgpt
Canonical inference weights: litgpt/lit_model.pth (~579 MiB, no optimizer). model.safetensors (~642 MiB) is the same LitGPT tensors.
Model
| Parameters | 303,353,856 |
| Context | 2048 |
| Vocab | 32,000 (arzlm-stem-32k-v2) |
| Layers / width | 24 / 1024 |
| Attention | 16 Q heads, 4 KV groups, head_dim 64, QK-Norm |
| MLP | SwiGLU, intermediate 2816 |
| RoPE | full, theta 10000 |
| Embeddings | tied |
Training
- 6,000,214,016 tokens, 45,778 optimizer steps, batch 64×2048
- Mixture 55% general / 20% math / 15% code / 10% science
- FineWeb-Edu sample-10BT (safe shard) + Cosmopedia-v2, FineMath-4+, Stack-Edu/SWH permissive, peS2o v2
- Muon hybrid + AdamW, cosine, warmup 800 steps, seed 42
- 1× NVIDIA A100-SXM4-40GB, PyTorch 2.11.0+cu128, LitGPT 0.5.13, BF16,
torch.compile
Validation (training packed val, not WikiText)
- loss 2.6909
- perplexity ~14.75
- by domain: general 3.300, math 1.800, code 1.472, science 2.955
Intended use
Research, tiny-LM experimentation, architecture / optimizer / data-mixture studies, educational reproduction of a 300M pretrain.
Limitations
300M base model, 2048 context, 6B tokens. Factually unreliable, weak math, repetition, template-like drift, incorrect code. Not aligned for assistants.
License
Apache-2.0 for these weights and the training code, with dataset attribution in the GitHub NOTICE. See that file for ODC-By / Common Crawl / Software Heritage notes.