EMO: Pretraining Mixture of Experts for Emergent Modularity
This page is an index for the model checkpoints released alongside EMO: Pretraining Mixture of Experts for Emergent Modularity. The repository at allenai/EMO does not host model weights — pick the checkpoint you want from the table below.
Released models
Main release
| Model | Description |
|---|---|
allenai/Emo_1b14b_1T | EMO — 1B-active / 14B-total MoE pretrained on 1T tokens + 50B-token midtraining anneal. The main model from the paper. |
Ablation: EMO at smaller scale
| Model | Description |
|---|---|
allenai/Emo_1b14b_130B | EMO trained on 130B tokens (Table 1 / Figure 11 ablation). Not midtrained. |
Architecture-matched standard MoE baselines
These share architecture and data with the EMO models above; only the training objective differs (no document-level expert pool constraint).
| Model | Description |
|---|---|
allenai/StdMoE_1b14b_1T | Standard MoE — Reg. MoE at 1T tokens in the paper. Same setup as Emo_1b14b_1T. |
allenai/StdMoE_1b14b_130B | Standard MoE — Reg. MoE at 130B tokens. Same setup as Emo_1b14b_130B. |
Memory-matched baselines (Figure 1)
Smaller models trained from scratch at fixed memory budgets, used as comparison points for EMO expert subsets.
| Model | Description |
|---|---|
allenai/Dense_1b_130B | Dense @ 8 — 1B dense decoder-only Transformer trained on 130B tokens. Active-parameter-matched with 8-expert subsets of the larger EMO/StdMoE models. |
allenai/StdMoE_1b4b_130B | Reg. MoE @ 32 — 1B-active / 4B-total standard MoE (32 routed experts) trained from scratch on 130B tokens. Memory-matched with 32-expert subsets. |
EMO-anneal ablation (Appendix B.4)
Tests whether modularity can be induced after pretraining by annealing a standard MoE under the EMO objective.
| Model | Description |
|---|---|
allenai/StdMoE_1b14b_1T_Preanneal | Standard MoE pretrained on 1T tokens, no annealing. The starting point for the EMO-anneal experiment. |
allenai/StdMoE_1b14b_1T_EmoAnnealed | EMO-anneal — StdMoE_1b14b_1T_Preanneal annealed for 50B tokens under the EMO document-level expert pool objective. |
Quick start
All checkpoints require trust_remote_code=True since they use custom modeling code from the ryanyxw/transformers fork. Replace model_id with the checkpoint you want from the table above.
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "allenai/Emo_1b14b_1T" # main EMO release
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
inputs = tokenizer(["Language modeling is "], return_tensors="pt", return_token_type_ids=False)
out = model.generate(**inputs, max_new_tokens=100, do_sample=True, temperature=1.0, top_p=0.7)
print(tokenizer.batch_decode(out, skip_special_tokens=True)[0])
Citation
@article{wang2026emo,
title = {EMO: Pretraining Mixture of Experts for Emergent Modularity},
author = {Wang, Ryan and Bhagia, Akshita and Min, Sewon},
year = {2026},
url = {https://arxiv.org/abs/2605.06663}
}
Links
- Paper: https://arxiv.org/abs/2605.06663
- Code: https://github.com/allenai/EMO
- Visualization: https://emovisualization.netlify.app