open-ogma-small
An 8.96M-parameter English sentence-embedding model with a fully permissive
license chain — MIT-licensed teachers, ODC-BY corpora, MIT weights. The
open-ogma family is the permissive successor to the axiotic/ogma-* family
(which is CC-BY-NC-4.0 due to its training-data mix).
- Architecture: 6-layer transformer trunk (d_model=256, d_output=256,
mean pooling, task tokens
[QRY]/[DOC]/[SYM]) + two projection heads:proj_small256→384 andproj_large256→1024. - Default encode path:
proj_small(384d) — the head used for the headline MTEB row. High-fidelity mode:proj_large(1024d) scores +0.011 mean / +0.028 retrieval over the default — opt in when output width is not a constraint. - Matryoshka: both heads and the trunk were trained with nested-slice losses at [32, 64, 128, 256] — truncate any output to its leading dims and re-normalise for smaller embeddings.
MTEB(eng, v2) — full 41-task benchmark
| output | dim | mean | Class | Clust | Pair | Rerank | Retr | STS | Summ |
|---|---|---|---|---|---|---|---|---|---|
proj_large (opt-in) | 1024 | 0.5843 | 0.7275 | 0.4332 | 0.7933 | 0.4549 | 0.4039 | 0.7840 | 0.2876 |
proj_small (default) | 384 | 0.5737 | 0.7135 | 0.4309 | 0.7911 | 0.4561 | 0.3760 | 0.7814 | 0.2897 |
| trunk (pre-head) | 256 | 0.5691 | 0.7230 | 0.4323 | 0.7965 | 0.4564 | 0.3422 | 0.7851 | 0.3015 |
Reference points measured with the identical harness (subprocess-per-task,
mteb 2.12/2.18; harness validated by reproducing minishlab/potion-base-8M
at 0.5328 vs its official 53.33): teacher bge-small-en-v1.5 (33M) scores
0.6368 and bge-large-en-v1.5 (335M) 0.6528 — this model retains ~90-92% of
the small teacher's mean at 27% of its parameters.
In-training small-mteb (20-task subset) best: 0.5857 at 88B tokens.
Recipe
Distilled for 100B tokens (bs=128 × seq 1024) from an ensemble of two MIT
teachers — BAAI/bge-small-en-v1.5 (384d) and BAAI/bge-large-en-v1.5
(1024d) — each supervised through its own projection head with equal loss
weight (0.5/0.5), on a 10.6M-document permissive blend of C4 and FineWeb-Edu
(contamination-flagged against MTEB test sets, 0.11% hits marked). Loss:
matryoshka-weighted distillation (α) + in-batch contrastive (β) on a
1.0→0.7/0.3 schedule; task-token mix {QRY 0.25, DOC 0.25, SYM 0.5}; AdamW
lr 5e-4 cosine, wd 0.01.
License chain (verified)
| component | license |
|---|---|
teacher BAAI/bge-small-en-v1.5 | MIT |
teacher BAAI/bge-large-en-v1.5 | MIT |
| corpus C4 (allenai/c4) | ODC-BY |
| corpus FineWeb-Edu (HuggingFaceFW) | ODC-BY |
| tokenizer (ogma sentencepiece, trained in-project) | MIT |
| these weights | MIT |
ODC-BY requires attribution for the source corpora, which this card provides. No non-commercial or share-alike terms anywhere in the chain.
Usage
# clone this repo, then:
from ogma_libre import OgmaLibre
model = OgmaLibre.from_repo("path/to/open-ogma-small")
embs = model.encode(["A quick brown fox."]) # (N, 384) default
embs = model.encode(["A quick brown fox."], head="large1024") # (N, 1024) best quality
embs = model.encode(["A quick brown fox."], head="base") # (N, 256) trunk
# Retrieval convention: encode queries with task="qry", documents with task="doc".
q = model.encode(["what is a fox?"], task="qry")
d = model.encode(["The fox is a small canid."], task="doc")
Files: model.safetensors (weights), config.json (architecture + heads),
tokenizer/ogma_sp.model (sentencepiece), ogma/ (vendored model code),
ogma_libre.py (loader).
Sibling
axiotic/open-ogma-micro — the 2.3M-parameter sibling (same recipe, micro trunk).