eranex-embedding-v1.0-base

eranex-embedding-v1.0-base is an efficient multilingual sentence encoder backbone based on a Bidirectional Mamba State Space Model (SSM) architecture.

Pre-trained on quality-filtered multilingual mC4 and aligned cross-lingually via OPUS-100 Translation Language Modeling (TLM), it serves as the foundational encoder for downstream classification, custom fine-tuning, and representation learning tasks.

Intended Use

[!NOTE] Foundational Backbone Notice: eranex-embedding-v1.0-base is an MLM pre-trained foundational backbone. Like standard MLM encoders (e.g. BERT, RoBERTa), raw representations exhibit embedding anisotropy prior to task-specific contrastive alignment.

• Recommended Applications: Downstream fine-tuning (intent classification, sentiment analysis, custom domain adaptation) and dense feature extraction.
• For Zero-Shot Semantic Search & RAG: Use Eranex-Embedding-Search (fine-tuned with contrastive objectives to calibrate embedding isotropy for cosine similarity search).

Model Highlights

• Linear O(N) Complexity: Powered by Bidirectional Mamba SSM layers, providing faster inference on longer sequences compared to quadratic Transformer attention mechanisms.
• Compact 128-Dimensional Representations: Engineered for low latency downstream architectures and efficient vector index storage.
• Cross-Lingual Representation Space: Shared multilingual representations pre-aligned across Turkish (tr), English (en), Spanish (es), German (de).
• Resource Efficient Footprint: Compact 14.5M parameter footprint designed for edge deployment, CPU/GPU inference, and browser runtimes (ONNX).

Evaluation Results

Evaluations were performed on holdout validation splits across target languages.

1. Masked Language Modeling (MLM)

2. Semantic Textual Similarity (STS) Spearman Correlation

3. Cross-Lingual Bitext Mining (Top-1 Accuracy)

Training Data & Methodology

eranex-embedding-v1.0-base was pre-trained using a targeted, quality-filtered data curation pipeline:

• Quality Filtering: Enforced text length constraints (150 to 5,000 characters) and minimum alphanumeric ratio (60%) to remove noise and unstructured content.
• Phase 1 (Multilingual MLM): Approximately 1,000,000 curated documents per language from the mC4 dataset.
• Phase 2 (TLM Alignment): Parallel sentence pairs from OPUS-100 across target language pairs.
• Optimization: Trained using bfloat16 mixed precision, 8-bit AdamW optimizer, and disk-backed memory-mapped streaming.

Usage

1. Feature Extraction

import torch
from transformers import AutoModel, PreTrainedTokenizerFast

repo_id = "Eranex-Tech/eranex-embedding-v1.0-base"

tokenizer = PreTrainedTokenizerFast.from_pretrained(repo_id)
model = AutoModel.from_pretrained(repo_id, trust_remote_code=True)
model.eval()

texts = [
    "Yapay zeka ve durum uzayi modelleri hizla ilerliyor.",
    "State space models and artificial intelligence are advancing rapidly.",
    "Der Hund spielt im Garten.",
    "El perro juega en el parque.",
]

inputs = tokenizer(texts, padding=True, truncation=True, max_length=1024, return_tensors="pt")

with torch.no_grad():
    embeddings = model(**inputs)

print("Representations shape:", embeddings.shape)

2. Custom Downstream Task Head

import torch
import torch.nn as nn
from transformers import AutoModel

class EranexCustomClassifier(nn.Module):
    def __init__(self, repo_id: str, num_classes: int):
        super().__init__()
        self.encoder = AutoModel.from_pretrained(repo_id, trust_remote_code=True)
        self.classifier = nn.Sequential(
            nn.Dropout(0.2),
            nn.Linear(128, num_classes)
        )

    def forward(self, input_ids, attention_mask=None):
        pooled_repr = self.encoder(input_ids, attention_mask=attention_mask)
        return self.classifier(pooled_repr)

Technical Specifications

Model Family

Attribution & License

• Lead Developer: Nazmi Yücel Çan (@N4YuC4)
• Organization: Eranex Tech
• License: Apache-2.0