tinymixtral — v3.0 MoE flagship

Flagship release. A Mixtral-style sparse MoE language model: ~477.5M total / ~276.1M active parameters (top-2 of 4 routed experts), trained from scratch on 8.05B tokens on a single GPU.

• HF repo: mikecovlee/tinymixtral

Model details

Design decisions (validated by a 400M-token ablation matrix):

• top-2 routing — top-1 was +12.2–19.5% worse in val PPL at 400M tokens; maintain top-2.
• QK-Norm kept — neutral at 400M tokens, retained as the trunk default.
• aux = 1e-3 — the auxiliary-loss coefficient is insensitive in the 4-expert / top-2 ablation.
• LR ladder 5e-4 / 5e-4 / 4e-4 / 3e-4 across the four segments (5e-4 was the sole significant factor, −5.3% val PPL in the matrix).

Training

Four ~2B-token segments, each with its own complete WSD schedule (warmup 700 → stable → linear decay over the final 10%); the segment boundary is the anneal point.

• bf16 weights + autocast, bf16 optimizer states (--bf16-optim), chunked cross-entropy, seed 42
• batch 48 × 1024 = 49,152 tokens/step, ~14.3k tok/s
• gradient clipping 1.0, AdamW (β=0.9, 0.95, wd 0.1), hourly keep-last-2 checkpoints

Training data

8.05B unique tokens, zero repetition, drawn from six public corpora and split into four strictly disjoint ~2B-token pools (no token is seen twice across the four segments). Validation always uses a held-out shard set that never appears in training.

Shares are rounded and sum to ~100%.

Evaluation

Measured with lm-evaluation-harness v0.4.12, 0-shot (cuda, bf16).

v3.0 (S4, 8.05B tokens) vs the previous releases:

Harness note. The mean above is the 8-task harness (includes BoolQ); the dense and instruction-tuned cards report a 7-task mean (excludes BoolQ). The two are not directly comparable.

Comparison with similar models

Same suite and settings, measured locally (lm-evaluation-harness v0.4.12, 0-shot, cuda, bf16):

SmolLM2-360M was trained on 4T tokens and Qwen3-0.6B on 36T tokens, versus 8.05B tokens (~500× and ~4500× less) for v3.0 on a single consumer GPU — the gap is primarily a data-budget difference, and v3.0 is competitive on BoolQ (0.615 vs 0.620 / 0.643).

Data efficiency: v3.0 matches the 1B MoE trained on the same 8B tokens using 2.5× fewer total parameters and 1.28× fewer active parameters (~1.3× fewer FLOPs per token).

Notes:

• Persistent weak spot: ARC-Challenge (0.255 vs 0.279 for the 1B MoE) — long-training sensitive, but it narrowed from −3.2pp at 6B to −2.4pp at 8B.
• v3.0 vs 6B (S3, mean 0.4257) is within noise; S4 is best on the most tasks.
• Early (10k–14k) and late (32k–34k) val-PPL bumps were transient low-entropy shard effects that self-healed (36k=16.31 → 38k=15.59); no data-pool change was needed.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("mikecovlee/tinymixtral", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("mikecovlee/tinymixtral")

inputs = tokenizer("The capital of France is", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Requires transformers and trust_remote_code=True (custom tinymixtral architecture). This is a base (pretrained) model; for chat use the instruction-tuned mikecovlee/tinymixtral-it.

Limitations

• Base model: no instruction tuning or safety alignment; not a chat model.
• Trained on only 8.05B tokens — knowledge tasks (MMLU, ARC-Challenge) are capacity/budget-bound and far below models trained on trillions of tokens.
• English-centric.

Family

• mikecovlee/tinymixtral — this model (477.5M MoE base, v3.0 flagship)
• mikecovlee/tinymixtral-it — MoE instruction-tuned (v3.0-it, 3M SFT)
• mikecovlee/tinymistral-276m — 276M dense base (iso-active ablation)
• mikecovlee/tinymistral-276m-it — 276M dense instruction-tuned (3M SFT)
• mikecovlee/tinymixtral-v1.1-1b — 1B MoE (earlier flagship)
• mikecovlee/tinymixtral-v1.1-0.5b — 0.5B-class MoE (data-quality ablation)
• mikecovlee/tinymixtral-v2.0-beta — shared-expert experiment (beta)
• mikecovlee/tinymixtral-v1.0 — legacy (C4)

Naming. The MoE family is published under tinymixtral; the dense 276M iso-active ablation companions use the tinymistral spelling. Both belong to the same project.

Citation

@misc{tinymixtral2026,
  title  = {TinyMixtral: a small Mixture-of-Experts language-model family},
  author = {Michael Lee},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/mikecovlee/tinymixtral}}
}

License

MIT (Copyright (C) 2026 Michael Lee).