Veyra2-Mango-30M-Base
Veyra2-Mango-30M-Base is a 30.7M-parameter Llama-like causal language model trained from scratch on approximately 30B tokens. It is a raw base model, not an instruction-tuned assistant. It is intended for research, benchmarking, continued pretraining, and small-model experimentation.
Model Details
| Property | Value |
|---|---|
| Parameters | 30,683,520 |
| Architecture | LlamaForCausalLM |
| Layers | 16 |
| Hidden size | 384 |
| Attention heads | 6 |
| KV heads | 2 |
| Head dim | 64 |
| Intermediate size | 1152 |
| Vocabulary size | 8192 |
| Context length used in training | 3072 |
| Activation | SwiGLU / SiLU |
| Normalization | RMSNorm |
| Attention | GQA |
| Positional encoding | RoPE |
| Weight tying | Tied input embeddings and LM head |
| Training tokens | Approximately 30B |
| Training precision | bfloat16 |
| Optimizer | AdamW |
Tokenizer
Special tokens:
<|endoftext|>: 0<|im_start|>: 1<|im_end|>: 2<|pad|>: 3
Training Data
The model was trained on a 30B-token pretraining mixture.
Stage 1 18,000,000,000 tokens 180 shards
Mixture:
dclm_baseline: 50%
finephrase: 20%
cosmopedia_v2: 10%
finemath_4plus: 10%
ultrafineweb_multistyle: 5%
ultrafineweb_qa: 5%
Stage 1.5 4,000,000,000 tokens 40 shards
This stage linearly transitions from the Stage 1 mixture to the Stage 2 mixture.
Stage 2 8,000,000,000 tokens 80 shards
Mixture:
finephrase: 30%
dclm_baseline: 30%
cosmopedia_v2: 18%
finemath_4plus: 10%
ultrafineweb_multistyle: 5%
ultrafineweb_qa: 5%
ultrachat: 2%
Training Summary
- Final step: 25,432
- Tokens seen: 30,000,000,000
- Tokens per step: 1,179,648
- Sequence length: 3072
- Last train loss: 2.5062
Usage
Notes on Generation
Veyra2-Mango-30M-Base is a raw base model. It is not instruction tuned and should not be expected to behave like a chat assistant. Open-ended generations can be unstable, repetitive, or factually unreliable. It's not a polished assistant.
Intended Use
This model is intended for:
- small language model research
- continued pretraining
- benchmarking
- experimentation with compact causal LMs
Limitations
- Not instruction tuned
- Not RLHF tuned
- Not safe for factual or high-stakes use without additional validation
- Can hallucinate names, citations, species, references, and technical claims
- Open-ended text may drift off-topic
- Context length during training was 3072 tokens
Citation
If you use this model, please cite the model repository:
veyra-ai/Veyra2-Mango-30M-Base