TinyMixtral 1B — Post-Trained
A 1.18B-parameter Mixture-of-Experts language model (351M active), post-trained on 1B tokens of educational and web text.
Model Details
| Property | Value |
|---|---|
| Architecture | Decoder-only Transformer with Sparse MoE |
| Total Parameters | 1,182,172,160 |
| Active Parameters | ~351M |
| Hidden Size | 1024 |
| Layers | 16 |
| Experts | 8 (top-2 routing) |
| Attention Heads | 16 query / 4 key-value (GQA) |
| Head Dimension | 64 |
| Intermediate Size | 2,816 (per expert) |
| Vocabulary | 32,000 |
| Context Length | 2,048 |
| Position Encoding | RoPE (theta=1e6) |
| Activation | SiLU |
| Norm | RMSNorm |
| Tied Embeddings | Yes |
Training
Pre-training (4B tokens):
- Data: FineWeb-Edu + Cosmopedia (89:11 blend)
- Schedule: WSD (warmup 2,000 → stable → decay)
- Peak LR: 7e-4
- Batch: 16 × 1,024 = 16,384 tokens/step
- Steps: 244,141
- Duration: ~102.5 hours
Post-training (1B tokens):
- Data: FineWeb-Edu + Cosmopedia continuation (second 1B slice)
- Schedule: WSD (warmup 2,000 → stable → decay)
- Peak LR: 2e-5
- Batch: 16 × 1,024
- Steps: 60,975
- Duration: ~25.9 hours
Benchmark Results
Harness (0-shot)
| Benchmark | Score |
|---|---|
| HellaSwag (acc_norm) | 0.313 |
| PIQA (acc) | 0.609 |
| Winogrande (acc) | 0.505 |
| ARC-Easy (acc_norm) | 0.410 |
| ARC-Challenge (acc_norm) | 0.272 |
| OpenBookQA (acc_norm) | 0.290 |
| BoolQ (acc) | 0.528 |
| LAMBADA (acc) | 0.195 |
IFEval (instruction-following)
| Model | inst_strict |
|---|---|
| 1B post-train | 0.2338 |
| v1.1 | 0.2182 |
SAMSum (Dialogue Summarization)
| Model | ROUGE-1 | ROUGE-2 | ROUGE-L |
|---|---|---|---|
| 1B (0-shot) | 9.83 | 0.50 | 7.85 |
| 1B (fine-tuned, 15ep) | 28.82 | 8.55 | 24.08 |
| T5-small (60M) | 35.7 | 13.4 | 31.4 |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"publish_posttrain/",
trust_remote_code=True,
torch_dtype="bfloat16",
device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("publish_posttrain/", legacy=False)
prompt = "The capital of France is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=20, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))
Limitations
- Data budget: Trained on only 4B tokens (pre-training) + 1B tokens (post-training). Comparable models use 100-1000× more data.
- Reasoning: Limited multi-step reasoning and mathematical capability.
- Hallucination: May generate plausible but incorrect facts.
- Context: Effective context is shorter than the 2,048-token window.
License
MIT License. See LICENSE for details.