MobileMoE-L (Base) Model Card

MobileMoE is a family of on-device Mixture-of-Experts (MoE) language models with sub-billion active parameters, designed to push the quality–efficiency Pareto frontier for on-device LLMs, including three model scales (S/M/L): 0.3B/0.5B/0.9B active parameters (1.3B/2.8B/5.3B total), with <3 GB INT4 weight footprints to fit in mobile DRAM. Each scale is released in three variants: a Base model (pre-training + mid-training), an SFT model (supervised fine-tuning), and a QAT model (quantization-aware training). You are currently in the MobileMoE-L-Base repository — the pre-trained 0.9B-active base model.

For the detailed technical report: 📝 MobileMoE: Scaling On-Device Mixture of Experts

For more versions, check out the 🤗 MobileMoE Collection

MobileMoE establishes a new Pareto frontier for on-device LLMs. Average benchmark accuracy, computed over 14 benchmarks spanning commonsense, knowledge, science, comprehension, and reasoning, is plotted against (a) per-token inference compute Finf = 2Nact (GFLOPs) and (b) total parameters Ntotal (B); in (b), x-axis tick labels show total params (B) | projected INT4 memory (GB).

Key Features

• A new Pareto frontier for on-device LLMs. Across 14 foundational benchmarks, MobileMoE matches or exceeds leading on-device dense LLMs at 2–4× fewer inference FLOPs, and matches or surpasses the state-of-the-art MoE OLMoE-1B-7B with up to 60% fewer parameters.
• Scaling-law-derived architecture. The architecture is derived from an on-device MoE scaling law that jointly optimizes under mobile memory and compute constraints, identifying an on-device sweet spot: moderate sparsity, with fine-grained experts and shared expert.
• Four-stage recipe. Pre-training → mid-training → instruction fine-tuning → INT4 quantization-aware training, all on open-source datasets.

Model Information

Model: MobileMoE-L-Base (pre-trained + mid-trained) Active Parameters: 922M Total Parameters: 5.3B Layers: 32 Model Dimension: 1280 Attention Heads: 20 KV Heads: 4 (GQA) Head Dimension: 64 Routed Experts: 60 (fine-grained, FFN hidden dim 640 each) Active Experts per Token: 4 (top-k sigmoid routing, with normalization) Shared Expert: 1, always on (FFN hidden dim 2560) Vocabulary Size: 128,256 Other Features: QK-Norm, tied input/output embeddings, RoPE (θ = 500,000) Input Modality: Text Output Modality: Text Languages: English Training Stages: Pre-training → mid-training Context Length: 8,192 tokens Precision: BF16 Model Developer: Meta Model Release Date: Aug 2026 License: MobileMoE is FAIR NC licensed

Results

All numbers below are for the base (pre-trained) models, re-evaluated under identical settings with greedy decoding using lm-eval; few-shot counts are given in parentheses after the benchmark name, and benchmarks shown without one are evaluated 0-shot.

Foundational benchmarks

Training

MobileMoE uses a four-stage recipe. This checkpoint is the output of stage 2 (mid-training).

MobileMoE four-stage training recipe: pre-training (PT) → mid-training (MT) → instruct supervised fine-tuning (SFT) → quantization-aware training (QAT) with INT4 precision.

How to use

MobileMoE uses a custom architecture (model_type: mobilemoe) that is not yet part of upstream transformers, so trust_remote_code=True is required. The modeling code ships in this repo (configuration_mobilemoe.py, modeling_mobilemoe.py).

Requirements

pip install "torch>=2.1" "transformers>=4.57" "safetensors>=0.4" "accelerate>=1.0"

Verified with the following versions:

For batch evaluation we recommend vLLM (≥ 0.10.2) with enforce_eager=True.

Text generation

This is a base model — it has no chat template and is not instruction-tuned. Prompt it with plain text continuation:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL_ID = "facebook/MobileMoE-L-Base"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    trust_remote_code=True,
    dtype=torch.bfloat16,
)
model.to("cuda" if torch.cuda.is_available() else "cpu")
model.eval()

prompt = "Why are open-source on-device language models great?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    input_ids=inputs["input_ids"],
    attention_mask=inputs["attention_mask"],
    max_new_tokens=64,
    do_sample=False,
    temperature=None,
    top_p=None,
    pad_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Known issues. Loading the tokenizer on transformers 4.57.6 prints a fix_mistral_regex=True warning. Please ignore it and do not set the flag, as MobileMoE uses the Llama-3 tokenizer whose default tokenization is already correct.

Citation

@article{chen2026mobilemoe,
  title={MobileMoE: Scaling On-Device Mixture of Experts},
  author={Chen, Yanbei and Huang, Hanxian and Chang, Ernie and Szwejbka, Jacob and Desai, Digant and Liu, Zechun and Chandra, Vikas and Krishnamoorthi, Raghuraman},
  journal={arXiv preprint arXiv:2605.27358},
  year={2026}
}

License

MobileMoE is distributed under the FAIR Noncommercial Research License.