mentee-embed-v1 — Trilingual Embeddings Trained From Scratch

mentee-embed-v1 is a compact 41M-parameter multilingual text embedding model supporting Arabic, English, and Urdu, trained entirely from scratch — no pretrained BERT or transformer base was used.

Trained in two stages on a single RTX 5090 GPU.

🌐 menteeai.org · 👤 syab.tech

Model Details

Training Recipe

Stage A — Masked Language Modeling

• Trained from random initialization on 1.1M trilingual sentences
• 6,000 steps, batch size 64, sequence length capped at 128
• Final val_mlm_loss ≈ 5.79 (expected for 41M params / 50K vocab / 62M tokens)

Stage B — Relational Knowledge Distillation

• Teacher: intfloat/multilingual-e5-base (768-dim)
• Method: Relational MSE — aligns the cosine similarity matrix of the student to match the teacher's, batch-by-batch
• Loss: rel_weight × MSE(C_student, C_teacher) + ce_weight × InfoNCE(anchor, positive)
• 6,000 steps, batch size 512, lr 2e-4 with cosine decay
• Final val acc@1 = 0.790 (128-way in-batch retrieval)

Training Data (1,110,575 triplets after dedup)

Benchmark Results

Evaluated against strong multilingual baselines on two protocols. Same test queries for every model.

Protocol A — In-batch Retrieval (pool ≈ 97 candidates)

Metrics: acc@1 / recall@5 / MRR@10

✅ mentee-embed-v1 beats all-MiniLM-L6-v2 across all metrics despite being trained from scratch. ✅ val acc@1 = 0.820 — beats paraphrase-MiniLM-L12-v2 (0.795) on the validation set. ✅ xling_en_ur acc@1 = 0.757 — strong cross-lingual English↔Urdu retrieval.

Protocol B — Corpus-pool Retrieval (15K docs per language, full ranking)

Metrics: MRR@10 · R@5 · R@100

The Protocol B gap vs baselines is expected: those models were trained on billions of pairs with hard negatives by large research teams. This model was trained on 1.1M pairs in a single GPU session. R@100 = 0.43–0.52 means the correct document is in the top 100 nearly half the time, which is meaningful for re-ranking pipelines.

Usage

Basic Sentence Embeddings

from tokenizers import Tokenizer
import torch
import torch.nn.functional as F
import numpy as np

# Load model
payload = torch.load("model.pt", map_location="cpu", weights_only=False)
from src.model import build_embedder
model = build_embedder(payload["encoder_config"], payload["vocab_size"])
model.load_state_dict(payload["state_dict"])
model.eval()

tok = Tokenizer.from_file("tokenizer.json")

def encode(texts):
    enc = tok.encode_batch(texts)
    ids = np.zeros((len(enc), max(len(e.ids) for e in enc)), dtype=np.int64)
    for i, e in enumerate(enc):
        ids[i, :len(e.ids)] = e.ids[:128]
    with torch.no_grad():
        emb = model(torch.from_numpy(ids))
    return F.normalize(emb, dim=-1).numpy()

texts = ["مرحبا بالعالم", "hello world", "دنیا میں خوش آمدید"]
embeddings = encode(texts)
print(embeddings.shape)  # (3, 384)

Semantic Similarity

from numpy import dot

emb = encode(["Information retrieval using embeddings", "How to search documents with vectors"])
similarity = dot(emb[0], emb[1])  # cosine similarity (vectors are normalized)
print(f"Similarity: {similarity:.3f}")

Limitations

• Protocol B corpus gap — open-domain retrieval over large corpora is weaker than web-scale models. Best used as a lightweight retriever or re-ranker in a pipeline.
• Training data scope — NLI + parallel translation data; may underperform on domain-specific retrieval (legal, medical, etc.)
• Sequence length — capped at 128 tokens; longer documents should be chunked.
• Script dependency — current model.pt format requires src/model.py from the mentee-embeddings repo to load. A sentence-transformers-compatible export is planned.

Author

Syed Syab Ahmad Shah is the founder of MenteE AI, building open-source multilingual NLP tools for underrepresented languages — with a focus on Arabic, Urdu, and English.

Links

• 🌐 Website: menteeai.org
• 💻 Code: github.com/MenteE-s/mentee-embeddings
• 🤗 Model: huggingface.co/MenteEAI/mentee-embed-v1

Citation

@misc{mentee-embed-v1,
  title  = {mentee-embed-v1: Trilingual Embeddings Trained From Scratch},
  author = {MenteE AI},
  year   = {2025},
  url    = {https://huggingface.co/MenteEAI/mentee-embed-v1}
}