Multilingual MiniLM-L12-v2 (Harrier-Distilled)
This model is a high-performance, lightweight sentence encoder aligned via embedding distillation to a top-performing cross-lingual teacher model from the MTEB leaderboard. Designed for high-throughput multilingual tasks, it delivers strong semantic retrieval capabilities while maintaining the speed and efficiency of a compact student architecture.
Model Details
| Parameter | Value |
|---|---|
| Student Backbone | sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 |
| Teacher Model | microsoft/harrier-oss-v1-270m |
| Best Epoch | 10 |
| Validation Cosine Similarity | 0.9377 |
| Best Validation MSE | 0.000195 |
| Learning Rate | 2e-05 |
| Batch Size | 16 |
| Max Sequence Length | 256 |
| License | Apache 2.0 |
Validation History
| Epoch | Train Loss | Val MSE | Val Cosine Similarity |
|---|---|---|---|
| 1 | 0.000493 | 0.000277 | 0.9114 |
| 2 | 0.000249 | 0.000234 | 0.9250 |
| 3 | 0.000224 | 0.000220 | 0.9297 |
| 4 | 0.000212 | 0.000211 | 0.9325 |
| 5 | 0.000203 | 0.000204 | 0.9346 |
| 6 | 0.000197 | 0.000200 | 0.9361 |
| 7 | 0.000193 | 0.000197 | 0.9370 |
| 8 | 0.000190 | 0.000195 | 0.9374 |
| 9 | 0.000188 | 0.000195 | 0.9377 |
| 10 | 0.000187 | 0.000195 | 0.9377 (Best) |
Installation
Because this model couples a transformer backbone with a trained projection layer, it requires the custom helper package:
pip install git+https://github.com/agentlans/proj-embed.git
Quick Start
from proj_embed import ProjEmbed
# Initialize the model (loads backbone and projection weights automatically)
model = ProjEmbed("agentlans/multilingual-MiniLM-L12-v2-harrier")
# Generate L2-normalized embeddings for a single text or a batch
embeddings = model.embed(["Hello world", "Greetings from ProjEmbed"])
print(embeddings.shape) # torch.Size([2, 640])
# Compute cosine similarity between two texts
similarity_score = model.similarity("Machine learning is fascinating.", "AI and deep learning are amazing.")
print(f"Similarity: {similarity_score:.4f}") # Similarity: 0.7600
Intended Use
- Cross-Lingual Retrieval: Fast semantic search across multiple languages with reduced compute overhead.
- Dataset Filtering: Efficient processing, cleaning, and filtering of large-scale translation and parallel corpora.
- Semantic Similarity: Rapid text clustering, deduplication, and ranking.
Limitations
- Dimension & Architecture Mismatch: Does not share the same hidden dimensions or architecture as the larger teacher model.
- Multilingual Nuance: Performance may vary on low-resource dialects or highly domain-specific terminology compared to major-language benchmarks.
License
Apache 2.0 like the base model