apertus-v1.5-swiss-embed-4.9b-bidir-langmoe
A Sentence Transformers retrieval model adapted from
andreasmartin/apertus-v1.5-swiss-embed-4.9b-bidir
by replacing the parent's final dense embedding projection with a learned sentence-level
SparseMixtureOfExperts projection.
Architecture
existing Sentence Transformers embedding parent
↓
all modules before the parent's final Dense
↓
SparseMixtureOfExperts
├── learned linear router
├── softmax
├── Top-2
└── weighted selected expert projections
↓
parent Normalize
This Swiss experiment uses 7 experts labelled
['de', 'en', 'fr', 'it', 'rm', 'gsw', 'shared']. The labels are experimental metadata; the MoE module itself
contains no language-specific logic.
No router temperature is used.
| Property | Value |
|---|---|
| Total parameters | 4,929,328,175 (4.929B) |
| Active parameters / sentence | 4,908,356,655 |
| Sparse MoE head parameters | 29,388,807 |
| Active head parameters / sentence | 8,417,287 |
| Router parameters | 28,679 |
| Parameters / expert | 4,194,304 |
| Experts | 7 |
| Active experts / sentence | 2 |
| Native embedding dimension | 1024 |
| Matryoshka dimensions | [1024, 768, 512, 256] |
| MoE training max length | 512 tokens |
| Inference max length | 1024 tokens |
| Sparse/reference max abs diff | 1.907e-06 |
Sentence Transformers usage
SparseMixtureOfExperts is currently exported as a repository-local candidate module. Until an
equivalent module is registered upstream in Sentence Transformers, loading this experimental
artifact requires trust_remote_code=True:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer(
"andreasmartin/apertus-v1.5-swiss-embed-4.9b-bidir-langmoe",
trust_remote_code=True,
)
queries = model.encode_query([
"Welche Aufgaben hat der Bundesrat?"
])
documents = model.encode_document([
"Le Conseil fédéral est l'autorité exécutive suprême de la Confédération suisse.",
"Der Nationalrat ist die grosse Kammer der Bundesversammlung.",
])
scores = model.similarity(queries, documents)
print(scores)
If SparseMixtureOfExperts becomes a registered Sentence Transformers module, the same conceptual
architecture can be serialized without repository-local Python code and loaded without
trust_remote_code=True.
Parent adaptation
| Property | Value |
|---|---|
| Existing embedding parent | andreasmartin/apertus-v1.5-swiss-embed-4.9b-bidir |
| Parent revision | 38e8ef280c560d9c871d20e38ebd9e353e2500f0 |
| Parent modules | ['Transformer', 'Pooling', 'Dense', 'Normalize'] |
| Replaced module | final Dense projection |
| Preserved output normalization | yes |
The parent was already an embedding model. This notebook does not create or modify a native backbone-level MoE LLM.
Training
The parent modules before the final Dense projection were frozen and their sentence representations cached. The original Dense projection initialized every expert, so the model starts from the parent's embedding transformation before experts specialize.
The retrieval objective is combined with:
- optional weak router supervision from the declared language labels;
- a load-balancing regularizer.
The declared language is not required at inference. Routing is computed from the sentence representation itself.
Training data:
eljuanina/VotingBooklets-Diamond-v1 is held-out evaluation-only.
Actual training set: 31,483 triplets.
See training_metadata.json for the exact training configuration,
router_diagnostics.json for held-out routing counts, and data_manifest.json for immutable
dataset revisions and processed-file hashes.
Internal retrieval diagnostics
These metrics are internal development diagnostics, not MTEB/MMTEB benchmark claims.
| Diagnostic | Dim | Accuracy@1 | Recall@10 | nDCG@10 | MRR@10 |
|---|---|---|---|---|---|
| Swiss monolingual | 1024 | 87.29% | 95.00% | 0.9109 | 0.8984 |
| Swiss monolingual | 512 | 87.29% | 95.00% | 0.9109 | 0.8985 |
| Swiss cross-lingual | 1024 | 33.75% | 99.38% | 0.7140 | 0.6183 |
| Swiss cross-lingual | 512 | 33.33% | 99.58% | 0.7121 | 0.6152 |
Monolingual evaluation queries: 480
Cross-lingual evaluation queries: 480
Router diagnostics
Recorded shared-expert Top-k selections in the held-out diagnostic: 15.
Limitations
- The adapter currently targets existing Sentence Transformers parents ending in
Dense -> Normalize. - Sparse routing occurs only at sentence-embedding projection level.
- Language labels are weak supervision for this experiment, not an architectural requirement.
- External MTEB/MMTEB/MIRACL and Swiss/domain-specific evaluation is required for comparative claims.
- The MoE head was trained on cached representations up to 512 tokens. The packaged parent retains 1024-token inference capability, but quality beyond the MoE training context length has not yet been separately validated.