Qwen2.5-0.5B-MoE-A0.3
A sparse mixture-of-experts model derived from Qwen/Qwen2.5-0.5B. Each token uses about 40% of the original MLP compute: 8 of 32 routed experts plus a shared expert in each MoE layer. Attention, embeddings and the tokenizer are the same as Qwen2.5-0.5B.
It is a stock Qwen2MoeForCausalLM, so it loads with plain transformers and needs no custom code:
from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained("bayliner1980/Qwen2.5-0.5B-MoE-A0.3B")
model = AutoModelForCausalLM.from_pretrained("bayliner1980/Qwen2.5-0.5B-MoE-A0.3B", dtype="bfloat16")
inputs = tok("The history of the printing press", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=50)
print(tok.decode(out[0], skip_special_tokens=True))
Architecture
| Layers | 24: layers 0-22 are MoE, layer 23 is a dense MLP (mlp_only_layers=[23]) |
| Routed experts | 32 per MoE layer, top-8, norm_topk_prob=True |
| Expert size | 128 |
| Shared expert | 768, sigmoid-gated |
| Active MLP compute | 37% of the dense model's in each MoE layer, 39.5% averaged over all layers |
| Active parameters | ~168M of 358M non-embedding parameters per token |
| Total parameters | 494.7M (bf16, ~990 MB) |
| Context length, vocabulary | Same as Qwen2.5-0.5B |
Evaluation
Perplexity uses 1024-token windows. Benchmarks are zero-shot and follow lm-evaluation-harness conventions (HellaSwag on 3,000 validation items; full sets for the others).
| WikiText-2 test ppl | C4 val ppl | HellaSwag (norm) | ARC-Easy (acc) | ARC-Challenge (norm) | WinoGrande | OpenBookQA (norm) | Mean acc_norm | |
|---|---|---|---|---|---|---|---|---|
| Qwen2.5-0.5B (dense) | 14.66 | 21.29 | 0.497 | 0.646 | 0.321 | 0.569 | 0.354 | 0.467 |
| This model | 19.20 | 28.97 | 0.440 | 0.555 | 0.265 | 0.519 | 0.308 | 0.408 |
Mean acc_norm averages acc_norm over all five tasks (ARC-Easy acc_norm: 0.508).
Changes from the previous version
The previous upload scored better on WikiText-2 (17.99) but worse on general web text (C4 ~35) and benchmarks (mean acc_norm 0.393). This version is more general-purpose. It uses slightly more compute per token than before (39.5% vs 37% of the MLP) because its last layer is dense.
Speed
The sparsity only saves compute with an inference engine that runs just the selected experts, such as
transformers' MoE kernels, vLLM or llama.cpp. At this model size attention and the embedding/LM head make up a
large share of the cost, so expect smaller end-to-end speedups than the MLP figures suggest.
Limitations
- Behind the dense model on reasoning and knowledge. It keeps most of the dense model's language-modelling quality, but loses about 0.06 mean benchmark accuracy. Expect factual errors and topic drift in long generations.
- Code and non-English text not evaluated. Quality there is likely weaker than on English prose.
- Base model, not instruction-tuned. It continues text; it does not follow chat instructions.
- Research artefact. It is an experiment in sparse conversion, not a model for production use.
Support
This model was built entirely on my own hardware, as an independent experiment. The results are promising, and
there's a lot more I'd like to try. Converting larger models, training for longer and testing more configurations
should all improve quality, but they need far more compute than my current hardware can provide.
With your support, I can rent cloud GPUs to run these experiments and release better and larger models. Any new
models will be shared openly here.
If you find this work useful or interesting, any contribution helps.
License
Apache 2.0, the same as the base model Qwen/Qwen2.5-0.5B.