gemma3moe-16x4-270m-starter
-ff8800.svg){kind=icon}
A ready-made Mixture-of-Experts initialization — 16 experts, top-4 — sparse-upcycled from
google/gemma-3-270m. This is a starting point for
your own training / fine-tuning, not a trained model: it is the standard copy-init upcycle, so at step
0 it is mathematically identical to the dense Gemma-3-270m. Load it, then train.
⚠️ Untrained. Out of the box it behaves exactly like
gemma-3-270m(verified: max logit diff0.000e+00). Its value is as a clean MoE init with more capacity to train into.
- 🧩 Architecture:
gemma3moe— Gemma-3 with each dense FFN replaced by a router + 16Gemma3MLPexperts (softmax top-4, renormalized). - 💾 Size: ~1.33B total / ~481M active (top-4). Full 262k vocabulary (not trimmed).
- ⚡ CPU-friendly: active params — not total — drive latency; runs on
llama.cpp. - 🔁 Self-contained: ships
modeling_gemma3_moe.py+auto_map, so it reloads withtrust_remote_code=True.
Why start from an upcycled MoE?
Sparse upcycling initializes each expert from the pretrained dense FFN, so you keep all of Gemma-3-270m's learned knowledge while adding capacity (16× the FFN parameters) that training can specialize. Copy-init makes the MoE exactly equal to the dense model at step 0 — a stable, well-behaved starting point, strictly better than training a randomly-initialized MoE from scratch.
| Total params | Active (top-4) | Vocab | |
|---|---|---|---|
| This checkpoint | ~1.33B | ~481M | 262k (full) |
google/gemma-3-270m (dense) | ~268M | ~268M | 262k |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
repo = "tardellirs/gemma3moe-16x4-270m-starter"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16)
# It's an INIT — fine-tune it on your data before using for generation.
Choose a different experts/top-k or a trimmed vocabulary, and serve on CPU via llama.cpp, using the
open-source toolkit that built this checkpoint:
➡️ github.com/tardellirs/gemma3moe-builder
# reproduce this checkpoint
python -m gemma3moe.upcycle --src google/gemma-3-270m --out ./starter --num-experts 16 --top-k 4
# fine-tune
python -m gemma3moe.train --init ./starter --train train.jsonl --val val.jsonl --out ./trained
Details
- Router initialized small but non-zero (
std=0.02) so gradients can differentiate the experts during training; all 16 experts are copies of the dense FFN at init. - Add a Switch-style load-balance aux loss (+ z-loss) when you train (see the toolkit's
train.pyanddocs/NOTES.md). - 18 layers, hidden 640, expert intermediate 2048.
License
Weights derive from Gemma-3 and are subject to the Gemma Terms of Use. Built with gemma3moe-builder (Apache-2.0).