bge-base-en-v1.5 fine-tuned on synthetic PubMed query–abstract pairs
Part of the study finetuned-embeddings-rag-eval.
Training
- Base model:
BAAI/bge-base-en-v1.5 - Data: 1,300 synthetic (query, abstract) pairs. Queries generated by
gemma3:4bfrom PubMed abstracts (ccdv/pubmed-summarization, train split). Pairs file:data/pubmed_training_pairs.jsonin the GitHub repo (first 1,300 used). - Loss: MultipleNegativesRankingLoss. 5 epochs, batch size 32, 100 warmup steps, lr 2e-5 (library default), mixed precision.
- Final model after 5 epochs (no checkpoint selection).
- Single training run on a Colab T4. sentence-transformers 5.5.1, transformers 5.16.1, torch 2.11.0.
Evaluation
Read first: on an objective metric (retrieved chunk comes from the source abstract) this model shows no statistically reliable improvement over untuned bge-base, and the automatic judge used in the table below agreed poorly with human labels (Cohen's kappa 0.2). 63 of the 90 test questions come from abstracts also used in training.
90 human-written queries over 7,627 chunks (2,000 PubMed abstracts, 400-char chunks, 50-char overlap).
Relevance judged automatically by cross-encoder/ms-marco-MiniLM-L-6-v2 (logit ≥ 0), not by human labels.
All models evaluated in the same run. Results file: experiments/pubmed_step11_scores.json.
| Model | Hit@3 | MRR | NDCG@10 |
|---|---|---|---|
| This model | 0.8667 | 0.7704 | 0.5609 |
| bge-base-en-v1.5 (no fine-tuning) | 0.8444 | 0.7222 | 0.5378 |
| nomic-embed-text (Ollama, no fine-tuning) | 0.7778 | 0.6519 | 0.5083 |
Paired tests, this model vs. untuned bge-base: McNemar (Hit@3) p = 0.6171; paired permutation (MRR) p = 0.0153. Untuned bge-base vs. nomic: p = 0.0771 (Hit@3), p = 0.0323 (MRR).
Most of the gain over nomic comes from the base model choice (provisional: no model used its recommended query prefix). Under this automatic judge, fine-tuning adds a small MRR gain. Under an objective metric (retrieved chunk comes from the source abstract), the pre-registered fine-tuned vs. untuned comparison is not significant (MRR 0.9722 vs. 0.9519, p = 0.299; on a 20,000-abstract corpus 0.7611 vs. 0.8019, p = 0.1158), and the automatic judge agreed poorly with human labels (Cohen's kappa 0.2, 60 items). Details in the GitHub README.
Limitations
- n = 90 queries.
- Pre-registered 3-seed check: mean MRR 0.7667 (SD 0.0064) vs. 0.7222 untuned, p = 0.0229; Hit@3 unchanged. Details in the GitHub README.
- Training queries are LLM-generated; relevance labels come from an automatic matcher.
- Queries were encoded without the bge query instruction prefix.
- Check the terms of the source dataset before commercial use.
Usage
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("rafiazarin/bge-base-pubmed-finetuned")
emb = model.encode(["effect of statins on cardiac events"], normalize_embeddings=True)