MiniMind-Embedding-MoE
A Mixture-of-Experts (MoE) text embedding model built on the MiniMind-3-MoE backbone (198M total / 64M active params), with the training pipeline fully aligned with Qwen3-Embedding. This is the MoE counterpart of MiniMind-Embedding-Dense.
📌 Introduction
This model uses the same Qwen3-Embedding recipe as the Dense version (causal attention + last-token pooling + InfoNCE with false-negative masking + MRL), but swaps the feed-forward layers for a 4-expert top-1 MoE. The goal is to study whether MoE architecture helps embedding quality at the small scale.
Core techniques
🏗️ Model Details
| Item | Value |
|---|---|
| Architecture | MiniMind-3-MoE (causal decoder-only, 4 experts top-1) |
| Total params | 198M |
| Active params | 64M (per token, 1 expert active) |
| Hidden size | 768 |
| Layers | 8 |
| Attention heads | 8 (query) / 4 (KV, GQA) |
| Vocab size | 6400 |
| Embedding output dim | 768 (MRL: truncatable to 512/256/128/64) |
| Max sequence length | 8192 |
| Pooling | last-token |
🎓 Training Details (3 Stages)
| Stage | Data | Loss | Steps | Final Loss | Status |
|---|---|---|---|---|---|
| 1. Weakly-supervised | T2Ranking triplets (90k) | Standard InfoNCE | 2,827 | ~0.45 | ✅ Done |
| 2. Supervised | T2Ranking-15 (340k, 15 hard negs) | InfoNCE + false-neg mask + MRL | 63,768 (3 epochs) | ~0.85 | ✅ Done |
| 3. Model merging | Last 5 Stage-2 checkpoints | SLERP | — | — | ✅ Done |
MoE Stage 1 (left) + Stage 2 (right), 3 epochs.
Dense vs MoE Training Loss
Note: MoE loss is consistently higher than Dense. This is expected — MoE routes tokens to different experts, making the InfoNCE contrastive task harder (each expert sees only a subset of the sequence's tokens). The higher train loss doesn't directly translate to worse embeddings, but the evaluation shows MoE underperforms Dense at this scale.
📊 Evaluation
C-MTEB STS (Spearman correlation)
| Task | Dense (64M) | MoE (198M-A64M) | Δ |
|---|---|---|---|
| ATEC | 0.265 | 0.201 | -0.064 |
| BQ | 0.379 | 0.297 | -0.082 |
| LCQMC | 0.631 | 0.582 | -0.049 |
| STSB | 0.637 | 0.607 | -0.030 |
| Average | 0.478 | 0.422 | -0.056 |
Key Finding: MoE underperforms Dense at small scale
💡 Surprising result: the MoE model (198M total / 64M active) underperforms the Dense model (64M) by 0.056 average STS. This is a counterintuitive but important finding:
- MoE fragmentation: with only 4 experts and top-1 routing, each expert sees only ~25% of tokens. For embedding (where the whole sentence's representation matters), this fragmentation hurts — no single expert gets a complete view of the sentence.
- Active params are the same: MoE activates 64M params per token (same as Dense's 64M), so it's not "bigger" in terms of compute — it's just more fragmented.
- Routing overhead: the router must learn to dispatch tokens, adding an optimization burden that competes with learning good representations.
Takeaway: for small embedding models, Dense > MoE. MoE benefits emerge at larger scales (Qwen3-Embedding uses MoE only at 4B+ params) where each expert is large enough to be expressive on its own.
🚀 Usage
from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim
model = SentenceTransformer("Muzian/minimind-embedding-moe")
documents = ["天空呈蓝色是因为阳光散射。", "红烧肉的做法是..."]
doc_embs = model.encode(documents, normalize_embeddings=True)
queries = ["天空为什么是蓝色的"]
query_embs = model.encode(queries, prompt_name="query", normalize_embeddings=True)
scores = cos_sim(query_embs, doc_embs)
Recommendation: for most use cases, prefer MiniMind-Embedding-Dense — it's smaller, faster, and scores higher. This MoE model is primarily for research/comparison.
⚠️ Limitations
- Same backbone limitations as Dense (vocab=6400, limited Stage-1 data)
- MoE underperforms Dense at this scale — see analysis above
- 198M params but only 64M active → inference is slower than Dense without quality gain
🙏 Acknowledgements
- Base: MiniMind by @jingyaogong
- Route: Qwen3-Embedding
- Data: T2Ranking
- Eval: MTEB
📄 License
Apache 2.0