MobileMoE-S (QAT) Model Card

MobileMoE is a family of on-device Mixture-of-Experts (MoE) language models with sub-billion active parameters, designed to push the quality–efficiency Pareto frontier for on-device LLMs, including three model scales (S/M/L): 0.3B/0.5B/0.9B active parameters (1.3B/2.8B/5.3B total), with <3 GB INT4 weight footprints to fit in mobile DRAM. Each scale is released in three variants: a Base model (pre-training + mid-training), an SFT model (supervised fine-tuning), and a QAT model (quantization-aware training). You are currently in the MobileMoE-S-QAT repository — the INT4 quantization-aware-trained 0.3B-active model, built on top of MobileMoE-S-SFT and intended for on-device deployment.

For the detailed technical report: 📝 MobileMoE: Scaling On-Device Mixture of Experts

For more versions, check out the 🤗 MobileMoE Collection

MobileMoE establishes a new Pareto frontier for on-device LLMs. Average benchmark accuracy, computed over 14 benchmarks spanning commonsense, knowledge, science, comprehension, and reasoning, is plotted against (a) per-token inference compute Finf = 2Nact (GFLOPs) and (b) total parameters Ntotal (B); in (b), x-axis tick labels show total params (B) | projected INT4 memory (GB). Accuracy is shown for the instruction-tuned models.

Key Features

• A new Pareto frontier for on-device LLMs. Across 14 foundational benchmarks, MobileMoE matches or exceeds leading on-device dense LLMs at 2–4× fewer inference FLOPs, and matches or surpasses the state-of-the-art MoE OLMoE-1B-7B with up to 60% fewer parameters.
• Scaling-law-derived architecture. The architecture is derived from an on-device MoE scaling law that jointly optimizes under mobile memory and compute constraints, identifying an on-device sweet spot: moderate sparsity, with fine-grained experts and shared expert.
• Four-stage recipe. Pre-training → mid-training → instruction fine-tuning → INT4 quantization-aware training, all on open-source datasets.

Model Information

Model: MobileMoE-S-QAT (INT4, quantization-aware trained) Active Parameters: 272M Total Parameters: 1.3B Layers: 20 Model Dimension: 768 Attention Heads: 12 KV Heads: 4 (GQA) Head Dimension: 64 Routed Experts: 60 (fine-grained, FFN hidden dim 384 each) Active Experts per Token: 4 (top-k sigmoid routing, with normalization) Shared Expert: 1, always on (FFN hidden dim 1536) Vocabulary Size: 128,256 Other Features: QK-Norm, tied input/output embeddings, RoPE (θ = 500,000) Input Modality: Text Output Modality: Text Languages: English Chat Template: Yes (end-of-turn token <|eot|>) Training Stages: Pre-training → mid-training → SFT → INT4 QAT Context Length: 8,192 tokens Precision: INT4 weights (symmetric, group size 32; FP16 scales); router stored and routing computed in FP32; BF16 norms and dense compute Projected INT4 Weight Memory: 0.68 GB Model Developer: Meta Model Release Date: Aug 2026 License: MobileMoE is FAIR NC licensed

Quantization

INT4 quantization-aware training is applied to the SFT checkpoint. Weights are quantized as

W̃g = sg · clamp(round(Wg / sg), −8, 7),    sg = 2·max(|Wg|) / (2^b − 1),   b = 4

where Wg is a contiguous group of 32 weights sharing one scale sg.

Note: Projected INT4 weight memory is a weight-only estimate. Packaged checkpoint files (model.safetensors) are 0.71 / 1.59 / 3.00 GB for S / M / L, including FP16 group scales, FP32 router weights, and BF16 norms/biases. Runtime memory is backend-dependent and additionally includes the KV cache, activations, and other overhead.

Results

All numbers below are for INT4-quantized models, re-evaluated under identical settings with greedy decoding using lm-eval; few-shot counts are given in parentheses after the benchmark name, and benchmarks shown without one are evaluated 0-shot. All three MobileMoE scales are the exact weights released in their respective QAT repositories, which include brief fine-tuning with self-identity beyond the QAT checkpoints in the technical report. This produces small differences against the report: averages of 43.9 / 52.7 / 58.1 here versus 44.0 / 52.5 / 57.8 for S / M / L in the report.

Foundational benchmarks (INT4)

Training

MobileMoE uses a four-stage recipe. This checkpoint is the output of stage 4 (INT4 QAT).

MobileMoE four-stage training recipe: pre-training (PT) → mid-training (MT) → instruct supervised fine-tuning (SFT) → quantization-aware training (QAT) with INT4 precision.

How to use

MobileMoE uses a custom architecture (model_type: mobilemoe) that is not yet part of upstream transformers, so trust_remote_code=True is required. The modeling code ships in this repo (configuration_mobilemoe.py, modeling_mobilemoe.py), and it handles INT4 dequantization automatically.

Requirements

pip install "torch>=2.1" "transformers>=4.57" "safetensors>=0.4" "accelerate>=1.0"

Verified with the following versions:

Chat

This model is instruction-tuned (QAT is applied on top of the SFT checkpoint), so use the chat template:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL_ID = "facebook/MobileMoE-S-QAT"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    trust_remote_code=True,
    dtype=torch.bfloat16,
)
model.to("cuda" if torch.cuda.is_available() else "cpu")
model.eval()

messages = [{"role": "user", "content": "Why are open-source on-device language models great?"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(
    input_ids=inputs["input_ids"],
    attention_mask=inputs["attention_mask"],
    max_new_tokens=1024,
    do_sample=False,
    temperature=None,
    top_p=None,
    pad_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

For multi-turn conversations, append each generated reply to messages with the assistant role. This ensures that each subsequent prompt includes the complete conversation history:

messages = []

for user_message in ["Who are you?", "Why are open-source on-device language models great?"]:
    messages.append({"role": "user", "content": user_message})
    input_ids = tokenizer.apply_chat_template(
        messages, add_generation_prompt=True, return_tensors="pt"
    ).to(model.device)
    outputs = model.generate(
        input_ids,
        attention_mask=torch.ones_like(input_ids),
        max_new_tokens=1024,
        do_sample=False,
        temperature=None,
        top_p=None,
        pad_token_id=tokenizer.eos_token_id,
    )
    reply = tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True).strip()
    messages.append({"role": "assistant", "content": reply})
    print(reply)

Known issues.

• Loading the tokenizer on transformers 4.57.6 prints a fix_mistral_regex=True warning. Please ignore it and do not set the flag, as MobileMoE uses the Llama-3 tokenizer whose default tokenization is already correct.
• The checkpoint stores qweight / weight_scale and creates dense weights on the first forward pass, so a "newly initialized" warning before then is expected.
• vLLM does not support this repository's INT4 format directly because its Transformers fallback replaces the custom layers and ignores the INT4 tensors. Convert the checkpoint to BF16 as described below, or use MobileMoE-S-SFT instead.

Exporting the INT4 Checkpoint to BF16

No conversion is needed for general use (e.g., with transformers) because the custom modeling code dequantizes the INT4 checkpoint to BF16 on the first forward pass. This section exports a numerically identical BF16 checkpoint to support other runtimes (e.g., vLLM) that bypass the custom loading logic and cannot use qweight / weight_scale directly.

import os, shutil, torch
from transformers import AutoModelForCausalLM, AutoTokenizer

SRC, DST = "facebook/MobileMoE-S-QAT", "./MobileMoE-S-QAT-bf16"

model = AutoModelForCausalLM.from_pretrained(SRC, trust_remote_code=True, dtype=torch.bfloat16)
model(torch.tensor([[128000]]))                 # first forward materializes INT4 -> BF16

state = {k: v for k, v in model.state_dict().items()
         if not (k.endswith("qweight") or k.endswith("_scale"))}
model.config.quantization = None                # mark the copy as plain BF16
model.save_pretrained(DST, state_dict=state, safe_serialization=True)
AutoTokenizer.from_pretrained(SRC).save_pretrained(DST)

# a trust_remote_code architecture needs its modeling files alongside the weights
for f in ("modeling_mobilemoe.py", "configuration_mobilemoe.py"):
    src_f = os.path.join(SRC, f) if os.path.isdir(SRC) else __import__(
        "huggingface_hub").hf_hub_download(SRC, f)
    shutil.copy2(src_f, os.path.join(DST, f))

Citation

@article{chen2026mobilemoe,
  title={MobileMoE: Scaling On-Device Mixture of Experts},
  author={Chen, Yanbei and Huang, Hanxian and Chang, Ernie and Szwejbka, Jacob and Desai, Digant and Liu, Zechun and Chandra, Vikas and Krishnamoorthi, Raghuraman},
  journal={arXiv preprint arXiv:2605.27358},
  year={2026}
}

License

MobileMoE is distributed under the FAIR Noncommercial Research License.