Zephyr RTOS documentation embeddings
all-MiniLM-L6-v2 fine-tuned on the Zephyr RTOS documentation, for retrieval over
that corpus.
- Comparison Space: https://huggingface.co/spaces/eoinedge/zephyrproject-embeddings-compare
- Index and pipeline: https://huggingface.co/eoinedge/zephyrproject
- Source: https://github.com/eoinjordan/zephyrproject-rag
What it is for
The base model is trained on general web text, where "binding" means a contract and "west" is a direction. This copy is adapted to Zephyr RTOS's vocabulary so those senses separate.
Use it as a drop-in replacement for the base model when embedding Zephyr RTOS documentation:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("eoinedge/zephyrproject-docs-embeddings")
vectors = model.encode(passages, normalize_embeddings=True)
Training
| Base | sentence-transformers/all-MiniLM-L6-v2 |
| Pairs | 6,079 mined from the docs' own heading/body structure |
| Loss | MultipleNegativesRankingLoss (in-batch negatives) |
| Epochs | 1 |
| Batch size | 32 |
| Held-out pairs | 400 |
No hand-labelled data and no synthetic questions from a generator — a section heading is a natural query for the body beneath it, and it is already written by the documentation authors.
Results, and how to read them
| Metric | Base | Tuned |
|---|---|---|
| recall@5 on held-out pairs | 0.63 | 0.86 |
That number flatters the model, and you should treat it with suspicion. The evaluation measures the same heading→body relationship the model was trained on, so it partly measures whether training converged rather than whether retrieval improved.
Spot-checking real questions gives a more mixed picture: some clearly improve, several are unchanged, and at least one regressed. Real questions are not phrased like section headings, which is exactly the gap this evaluation does not cover.
The comparison Space runs both models on the same query so you can judge for yourself rather than trusting the headline.
Limitations
- One epoch, one seed, no hyperparameter search.
- Evaluation is aligned with the training objective, as above.
- Trained on Zephyr RTOS documentation only — it will not help on other corpora and may be worse than the base model there.
- This adapts the retriever, not a generator. It does not teach a coding model anything about Zephyr RTOS; it changes which passages that model is handed.
Licence
Apache-2.0, matching the base model.