SkillRet-Embedding-0.6B
This is a sentence-transformers model fine-tuned for AI agent skill retrieval. Given a natural-language user request, the model retrieves relevant agent skills from a large skill library.
The model is fine-tuned from Qwen/Qwen3-Embedding-0.6B on the SkillRet benchmark training split using contrastive learning (MultipleNegativesRankingLoss).
📄 Technical report: SkillRet: A Large-Scale Benchmark for Skill Retrieval in LLM Agents (arXiv:2605.05726)
Usage
Sentence Transformers
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("ThakiCloud/SkillRet-Embedding-0.6B", trust_remote_code=True)
query_prompt = "Instruct: Given a skill search query, retrieve relevant skills that match the query\nQuery: "
queries = [
query_prompt + "Help me set up a CI/CD pipeline for my Python project"
]
skills = [
"ci-cd-setup | Configure continuous integration and deployment pipelines ...",
"python-debugging | Debug Python applications using pdb and logging ...",
]
q_emb = model.encode(queries, normalize_embeddings=True)
s_emb = model.encode(skills, normalize_embeddings=True)
similarities = q_emb @ s_emb.T
print(similarities)
Training Details
- Base model: Qwen3-Embedding-0.6B (0.6B parameters)
- Training data: SkillRet benchmark training split (127,190 query–skill pairs from 63,259 queries and 10,123 skills)
- Loss: MultipleNegativesRankingLoss (InfoNCE) with cross-GPU negative sharing
- Hardware: 4× NVIDIA B200 GPUs (DDP)
- Effective batch size: 384 (96 per device × 4 GPUs)
- Max sequence length: 8,192 tokens
- Learning rate: 2e-5
- Epochs: 1
- Training time: ~6 hours
- Precision: BF16
Evaluation Results
Evaluated on the SkillRet benchmark test split (4,392 queries, 6,006 skills).
| Metric | @5 | @10 | @15 |
|---|---|---|---|
| NDCG | 0.7902 | 0.8112 | 0.8187 |
| Recall | 0.8238 | 0.8774 | 0.9016 |
| Completeness | 0.7083 | 0.7894 | 0.8265 |
| MAP | 0.7409 | 0.7538 | 0.7570 |
| Precision | 0.2591 | 0.1401 | 0.0962 |
Intended Use
This model is designed for retrieving agent skills given natural-language user requests. It is part of the SkillRet benchmark submission for evaluating skill retrieval systems for AI agents.
Limitations
- Optimized for English-language queries and agent skills.
- Performance may vary on domains outside the SkillRet benchmark distribution.
- The model retrieves skills but does not execute them.
Framework Versions
- Python: 3.10.12
- Sentence Transformers: 5.4.1
- Transformers: 5.5.4
- PyTorch: 2.7.1+cu128
Citation
If you use this model or the SkillRet benchmark, please cite:
@article{kang2026skillret,
title = {SkillRet: A Large-Scale Benchmark for Skill Retrieval in LLM Agents},
author = {Kang, Ryangkyung and Cho, Hongcheol and Kim, Youngeun},
journal = {arXiv preprint arXiv:2605.05726},
year = {2026},
url = {https://arxiv.org/abs/2605.05726}
}
Paper: https://arxiv.org/abs/2605.05726
Paper (measurements behind the quantized variants and the student frontier): Where Post-Training Quantization Breaks Text Embedders: A Measured Map Across Four Embedder Families, arXiv:2609.16391
@article{han2026ptqembedders,
title = {Where Post-Training Quantization Breaks Text Embedders: A Measured Map Across Four Embedder Families},
author = {Han, Hyojung},
journal = {arXiv preprint arXiv:2609.16391},
year = {2026}
}