mentee-embed-v4

A 41M-parameter trilingual text embedding model trained entirely from scratch — no pretrained backbone, no BERT, no RoBERTa. Random initialization only.

v4 adds mMARCO Arabic retrieval data (500K triplets) and a third distillation round with hard negatives, yielding significant improvements over v3 across all languages and benchmarks.

Developed by Syed Syab Ahmad Shah and Team MenteE AI (menteeai.org) as part of an ongoing research effort to build competitive multilingual embeddings from the ground up for Arabic, English, and Urdu.

Key Facts

What "From Scratch" Means

Most embedding models fine-tune an existing pretrained encoder (BERT, RoBERTa, MPNet). mentee-embed-v4 does not. We:

1. Trained a custom BPE tokenizer on Arabic, English, and Urdu text
2. Initialized a 12-layer Transformer with random weights
3. Ran masked language modeling pretraining (50K steps on trilingual corpus)
4. Applied three-round contrastive distillation with hard negative mining

No pretrained checkpoint was used at any stage.

v4 vs v3 Improvements

Key improvements come from:

• mMARCO Arabic — 500K Arabic retrieval triplets added to training
• 50K MLM steps — 6x more pretraining than v3 (8K steps)
• 3 distillation rounds — extra hard-negative round for better discrimination

Full Benchmark Results

All baselines evaluated under identical conditions on the same hardware (RTX 5090).

MenteE Embed Bench — Custom EN/AR/UR Pilot

121 queries · 18 domains · dialect Arabic · Roman Urdu · hard negatives

Per-Language Breakdown

Detailed Metrics (NDCG@10, Recall@1, Recall@5, Recall@10)

Per-Domain MRR@10 (v4)

Protocol A — In-batch Retrieval (MIRACL)

acc@1 / MRR@10

v4 achieves strong MIRACL scores despite being 6.8x smaller than mpnet-base. MIRACL AR (0.874) is within 3% of mpnet-base (0.898).

Protocol C — MS-MARCO Corpus Retrieval (10K passages)

STS-B (Semantic Textual Similarity)

Speed & Efficiency

Measured on RTX 5090.

v4 is the fastest model tested — 3.5x faster than mpnet-base, 45% faster than MiniLM-L12.

Batch Size Scaling

Precision Formats

Efficiency Metrics (Per Parameter)

Radar — v4 vs mpnet-base

Normalized comparison across all evaluation dimensions:

Honest Assessment

Where v4 Wins

• Speed — fastest embedding model tested (18,115 sents/sec)
• Cost — cheapest per billion sentences ($7.15)
• MIRACL AR — 0.874 vs mpnet's 0.898 (within 3%) despite 6.8x fewer params
• MS-MARCO — 0.706, competitive with models 3-7x larger
• Efficiency — best MRR per parameter of all models tested

Where v4 Loses

• Custom bench — 0.252 vs mpnet's 0.661 (2.4x gap)
• English — 0.369 vs MiniLM-L6's 0.873
• STS-B — 0.725 vs mpnet's 0.868 (16% gap)
• Arabic dialect — 0.095 (struggles with informal Arabic)
• Urdu — 0.261 vs mpnet's 0.611

Verdict

v4 is a strong proof-of-concept — it proves you can train a competitive embedding model from scratch. The MIRACL AR score (0.874) is within 3% of mpnet-base (0.898) despite being 6.8x smaller.

For production use, v4 works well for:

• Arabic retrieval (MIRACL-scale tasks)
• High-throughput, cost-sensitive applications
• Resource-constrained environments (175 MB VRAM)

v4 is not yet suitable for:

• Domain-specific English retrieval
• Dialectal Arabic / informal Urdu
• Tasks requiring STS-level semantic similarity

Training Data

Training Pipeline

Stage 1 — MLM Pretraining (50K steps)
  Random init → masked language modeling on trilingual corpus
  50,000 steps · batch=64 · vocab=50K BPE

Stage 2 — Teacher Encoding
  Teacher: intfloat/multilingual-e5-base (768-dim)
  Encode all 2.6M triplets to soft labels

Stage 3 — Distillation Round 1 (no hard negatives)
  InfoNCE contrastive + relational distillation
  4,000 steps · batch=512 · temp=0.05

Stage 4 — Hard Negative Mining Round 1
  GPU-accelerated top-5 mining across full 2.6M corpus

Stage 5 — Distillation Round 2 (with hard negatives)
  Same objective + mined hard negatives per anchor
  10,000 steps · batch=512 · temp=0.05

Stage 6 — Hard Negative Mining Round 2 (on improved model)
  Re-mine with the better round-2 model

Stage 7 — Distillation Round 3 (final polish)
  Train on round-2 mined negatives
  15,000 cumulative steps · batch=512 · temp=0.05

Usage

# pip install torch transformers tokenizers huggingface_hub
from transformers import AutoModel, AutoTokenizer

tok   = AutoTokenizer.from_pretrained("MenteEAI/mentee-embed-v4", trust_remote_code=True)
model = AutoModel.from_pretrained("MenteEAI/mentee-embed-v4",   trust_remote_code=True)

sentences = [
    "Hello, how are you?",
    "مرحبا، كيف حالك؟",
    "ہیلو، آپ کیسے ہیں؟"
]

embeddings = model.encode(sentences, tokenizer=tok)
print(embeddings.shape)  # torch.Size([3, 384])

trust_remote_code=True is required — standard for custom-architecture models on HuggingFace. The code runs entirely on your machine.

Similarity search

from transformers import AutoModel, AutoTokenizer

tok   = AutoTokenizer.from_pretrained("MenteEAI/mentee-embed-v4", trust_remote_code=True)
model = AutoModel.from_pretrained("MenteEAI/mentee-embed-v4",   trust_remote_code=True)

query    = model.encode(["What is machine learning?"], tokenizer=tok)
passages = model.encode([
    "Machine learning is a subset of artificial intelligence.",
    "The weather today is sunny.",
    "تعلم الآلة هو فرع من فروع الذكاء الاصطناعي.",
], tokenizer=tok)

scores = query @ passages.T
print(scores)  # tensor([[0.81, 0.60, 0.79]])

Limitations

• Custom benchmark gap — v4 scores 0.252 vs 0.661 for mpnet-base on the custom bench; the gap reflects the difference between 2.6M training pairs and web-scale data
• Arabic and Urdu lag behind English on the custom bench, though MIRACL scores are strong (AR 0.874, UR 0.572)
• Vocabulary limited to 50K tokens trained on ~2.6M sentences — rare scripts and dialects may tokenize poorly
• Not evaluated on MTEB full suite yet

Citation

@misc{mentee-embed-v4-2026,
  title   = {How Far Can Multilingual Text Embeddings Be Trained From Scratch?
             A Compute-Efficient Study of Arabic, English, and Urdu},
  author  = {Shah, Syed Syab Ahmad and Team MenteE AI},
  year    = {2026},
  url     = {https://huggingface.co/MenteEAI/mentee-embed-v4},
  note    = {MenteE AI. Apache-2.0 License}
}

About MenteE AI

Built by Syed Syab Ahmad Shah and Team MenteE AI. menteeai.org · syab@menteeai.org