Veyra2-Blueberry-5M-Base
Veyra2-Blueberry-5M-Base is a 5M-parameter Llama-like causal language model trained from scratch on approximately 30B tokens. It is a raw base model, not an instruction-tuned assistant. It is intended for research, benchmarking, continued pretraining, and small-model experimentation.
Model Details
| Property | Value |
|---|---|
| Parameters | 4,984,192 |
| Architecture | Qwen3ForCausalLM |
| Layers | 5 |
| Hidden size | 256 |
| Attention heads | 4 |
| KV heads | 2 |
| Head dim | 64 |
| Intermediate size | 768 |
| Vocabulary size | 8192 |
| Context length used in training | 2048 |
| Activation | SwiGLU / SiLU |
| Normalization | RMSNorm |
| Attention | GQA |
| Positional encoding | RoPE |
| Weight tying | Tied input embeddings and LM head |
| Training tokens | Approximately 30B |
| Training precision | bfloat16 |
| Optimizer | Muon |
Tokenizer
Special tokens:
<|eos|>: 0<|bos|>: 1<|im_start|>: 2<|im_end|>: 3<|pad|>: 4
Training Data
The model was trained on a 30B-token pretraining mixture.
30,000,000,000 tokens 120 shards
Mixture:
fineweb_edu: 40%
finephrase: 20%
dclm_baseline: 20%
finemath_4plus: 20%
Training Summary
- Final step: 14,305
- Tokens seen: 30,000,000,000
- Tokens per step: 2.097M
- Sequence length: 2048
- Last train loss: 2.95
Usage
Notes on Generation
Veyra2-Blueberry-5M-Base is a raw base model. It is not instruction tuned and should not be expected to behave like a chat assistant. Open-ended generations can be unstable, repetitive, or factually unreliable. It's not a polished assistant.
Intended Use
This model is intended for:
- small language model research
- continued pretraining
- benchmarking
- experimentation with compact causal LMs
Limitations
- Not instruction tuned
- Not RLHF tuned
- Not safe for factual or high-stakes use without additional validation
- Can hallucinate names, citations, species, references, and technical claims
- Open-ended text may drift off-topic
- Context length during training was 2048 tokens
Citation
If you use this model, please cite the model repository:
veyra-ai/Veyra2-Blueberry-5M-Base