Veyra2-Apricot-50M-Base
Veyra2-Apricot-50M-Base is a 49.3M-parameter Llama-like causal language model trained from scratch on approximately 20B tokens.
It is a raw base model, not an instruction-tuned assistant. It is intended for research, benchmarking, continued pretraining, and small-model experimentation.
Model Details
| Property | Value |
|---|---|
| Parameters | 49,303,040 |
| Architecture | Llama-like causal LM |
| Layers | 16 |
| Hidden size | 512 |
| Attention heads | 8 |
| KV heads | 2 |
| Head dim | 64 |
| Intermediate size | 1408 |
| Vocabulary size | 8192 |
| Context length used in training | 1024 |
| Activation | SwiGLU / SiLU |
| Normalization | RMSNorm |
| Attention | GQA with QKV-Norm |
| Positional encoding | RoPE |
| Weight tying | Tied input embeddings and LM head |
| Training tokens | Approximately 19.99B |
| Training precision | bfloat16 |
| Optimizer | Muon |
Tokenizer
Special tokens:
<|bos|>: 0<|eos|>: 1<|pad|>: 2<|unk|>: 3<|im_start|>: 4<|im_end|>: 5
Training Data
The model was trained on a 20B-token pretraining mixture.
- FineWeb-Edu sample-10BT: 45%
- Cosmopedia-v2: 30%
- DCLM-Edu: 15%
- FineMath finemath-3plus: 10%
Training Summary
- Final step: 9532
- Tokens seen: 19,990,052,864
- Tokens per step: 2,097,152
- Global batch size: 2048
- Sequence length: 1024
- Last train loss: 2.5062
- Final eval loss: 2.4978
- Final eval perplexity: 12.16
Evaluation
SciCloze Eval
- SciCloze-900 accuracy: 47.56%
- SciCloze-900 correct: 428 / 900
SciCloze-900 repository: veyra-ai/SciCloze-900
SciCloze-900 subject breakdown:
- Biology: 53.67%
- Physics: 45.67%
- Chemistry: 43.33%
Other Evals
Zero-shot results:
Open SLM Bench verified:
- Average: 38.81%
- PIQA acc_norm: 62.13%
- ARC-Easy acc_norm: 42.47%
- ARC-Challenge acc_norm: 23.29%
- HellaSwag acc_norm: 31.28%
- ArithMark-3.0 acc: 31.60%
Additional local evaluations:
- Winogrande acc: 50.12%
- OpenBookQA acc_norm: 28.80%
- BoolQ acc: 60.67%
- SciQ accuracy: 72.70%
- SciQ normalized accuracy: 64.20%
Usage
This model uses custom Transformers remote code because it includes QKV-Norm and a custom architecture definition.
Notes on Generation
Veyra2-Apricot-50M-Base is a raw base model. It is not instruction tuned and should not be expected to behave like a chat assistant. Open-ended generations can be unstable, repetitive, or factually unreliable. It's not a polished assistant.
Intended Use
This model is intended for:
- small language model research
- continued pretraining
- benchmarking
- science-oriented cloze and multiple-choice evaluation
- experimentation with compact causal LMs
Limitations
- Not instruction tuned
- Not RLHF tuned
- Not safe for factual or high-stakes use without additional validation
- Can hallucinate names, citations, species, references, and technical claims
- Open-ended text may drift off-topic
- Context length during training was 1024 tokens
Citation
If you use this model, please cite the model repository:
veyra-ai/Veyra2-Apricot-50M-Base