tinymixtral-v1.1-1b — 1B MoE
Earlier flagship (before v3.0). A wider sparse MoE: ~1.18B total / ~352M active params (top-2 of 8 routed experts), trained on a SmolLM-inspired blend.
Model details
| Parameter | Value |
|---|---|
| hidden_size | 1024 |
| num_layers | 16 |
| Attention | GQA (16 heads / 4 KV heads) |
| Head dim | 64 |
| RoPE theta | 1,000,000 |
| Norm | RMSNorm |
| Experts | 8 routed (top-2) |
| Expert FFN | SwiGLU, intermediate = 2816 |
| Vocab size | 32,000 |
| Max position | 2,048 |
| Total params | ~1.18B |
| Active params | ~352M |
Training
| Phase | Data | LR | Tokens | Steps | Time | End loss |
|---|---|---|---|---|---|---|
| Pretrain | SmolLM blend (89:11) | 7e-4 | 4B | 244,141 | ~102.5 h | 1.9* |
| Continuation | non-overlapping new 4B, same 89:11 | 4.2e-4 | 4B | 121,921 | 60.4 h | 1.80 |
* Decay-phase loss plateaued at ~1.9 then spiked to 3.4 in the final ~300 steps as the data stream wrapped back to the start of the corpus.
Data: FineWeb-Edu + Cosmopedia v2 (89:11). Pretrain batch 16 × 1024 (16,384 tokens/step) → 4B tokens in 244,141 steps; warmup 2,000 steps. The continuation added 4B tokens in 121,921 steps (~32k tokens/step).
Evaluation
Measured with lm-evaluation-harness v0.4.12, 0-shot (cuda, bf16).
| Task | Metric | 1B MoE 4B | 1B MoE 8B |
|---|---|---|---|
| HellaSwag | acc_norm | 0.311 | 0.329 |
| PIQA | acc | 0.620 | 0.630 |
| WinoGrande | acc | 0.510 | 0.523 |
| ARC-Easy | acc | 0.463 | 0.479 |
| ARC-Challenge | acc_norm | 0.273 | 0.279 |
| OpenBookQA | acc_norm | 0.288 | 0.306 |
| BoolQ | acc | 0.548 | 0.620 |
| LAMBADA | acc | 0.200 | 0.234 |
| Mean | — | 0.402 | 0.425 |
Doubling pretrain tokens improved every task (BoolQ +7.2pp, LAMBADA +3.5pp, mean +2.3pp).
Few-shot: HellaSwag 10-shot acc_norm 0.312 · WinoGrande 5-shot acc 0.524 · ARC-Easy 25-shot acc_norm 0.468 · ARC-Challenge 25-shot acc_norm 0.262.
Comparison with similar models
Same suite and settings, measured locally (lm-evaluation-harness v0.4.12, 0-shot, cuda, bf16):
| Task | Metric | 1B MoE (8B) | SmolLM2-360M | Qwen3-0.6B |
|---|---|---|---|---|
| HellaSwag | acc_norm | 0.329 | 0.563 | 0.473 |
| PIQA | acc | 0.630 | 0.719 | 0.673 |
| WinoGrande | acc | 0.523 | 0.587 | 0.563 |
| ARC-Easy | acc | 0.479 | 0.705 | 0.609 |
| ARC-Challenge | acc_norm | 0.279 | 0.383 | 0.340 |
| OpenBookQA | acc_norm | 0.306 | 0.372 | 0.316 |
| BoolQ | acc | 0.620 | 0.620 | 0.643 |
| LAMBADA | acc | 0.234 | 0.532 | 0.401 |
SmolLM2-360M was trained on 4T tokens and Qwen3-0.6B on 36T tokens, versus 8B tokens for the 1B MoE on a single consumer GPU; the gap is primarily a data-budget difference.
Instruction following (IFEval)
IFEval (instruction-level loose accuracy), --apply_chat_template:
| Model | IFEval inst-level loose |
|---|---|
| 1B MoE 4B | 0.222 |
| 1B MoE 5B (post-train) | 0.234 |
| 1B MoE 8B (continuation) | 0.221 |
Instruction-following stays flat (~0.22) — more pretraining tokens do not improve it.
Post-training (negative result)
Post-trained from the lowest-loss pretrain checkpoint with the v1.1 recipe (Wiki + Cosmopedia v2 50:50, 1B tokens, lr 2e-5, warmup 300, WSD, 60,975 steps, ~26 h). Loss fell 2.98 → 1.8 but downstream metrics were unchanged:
| Task | Pretrain (4B) | Post-train (5B) |
|---|---|---|
| HellaSwag (acc_norm) | 0.311 | 0.313 |
| PIQA (acc) | 0.620 | 0.623 |
| WinoGrande (acc) | 0.510 | 0.505 |
| ARC-Easy (acc) | 0.463 | 0.465 |
| ARC-Challenge (acc_norm) | 0.273 | 0.272 |
| OpenBookQA (acc_norm) | 0.288 | 0.290 |
| BoolQ (acc) | 0.548 | 0.528 |
| LAMBADA (acc) | 0.200 | 0.195 |
At this scale, low-LR post-training on a knowledge blend lowers the loss but does not transfer to task ability.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("mikecovlee/tinymixtral-v1.1-1b", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("mikecovlee/tinymixtral-v1.1-1b")
Requires transformers and trust_remote_code=True (custom tinymixtral architecture).
Limitations
- Base model only: no instruction tuning or safety alignment.
- Trained on 8B tokens; English-centric. Superseded by the v3.0 flagship.
Family
mikecovlee/tinymixtral— 477.5M MoE base (v3.0 flagship)mikecovlee/tinymixtral-it— MoE instruction-tuned (v3.0-it, 3M SFT)mikecovlee/tinymistral-276m— 276M dense base (iso-active ablation)mikecovlee/tinymistral-276m-it— 276M dense instruction-tuned (3M SFT)mikecovlee/tinymixtral-v1.1-1b— this model (1B MoE, earlier flagship)mikecovlee/tinymixtral-v1.1-0.5b— 0.5B-class MoE (data-quality ablation)mikecovlee/tinymixtral-v2.0-beta— shared-expert experiment (beta)mikecovlee/tinymixtral-v1.0— legacy (C4)
Naming. The MoE family is published under
tinymixtral; the dense 276M iso-active ablation companions use thetinymistralspelling. Both belong to the same project.
Citation
@misc{tinymixtralv1.11b2026,
title = {TinyMixtral: a small Mixture-of-Experts language-model family},
author = {Michael Lee},
year = {2026},
howpublished = {\url{https://huggingface.co/mikecovlee/tinymixtral-v1.1-1b}}
}
License
MIT (Copyright (C) 2026 Michael Lee).