MA-Retriever
MA-Retriever is the dense retriever released with MetaSyn. It fine-tunes
BAAI/bge-large-en-v1.5 for retrieving articles linked to the published
included-study lists of systematic reviews, scoping reviews, and
meta-analyses.
Resources
- Paper: arXiv:2606.17041
- Dataset: THUIR/MetaSyn
- Code and evaluator: THUIR/MetaSyn
Use
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("BFTree/MA-Retriever")
def encode_protocol(research_question, population, intervention_or_exposure,
comparison, outcome):
fields = [
("Research Question", research_question),
("Population", population),
("Intervention or Exposure", intervention_or_exposure),
("Comparison", comparison),
("Outcome", outcome),
]
protocol = ". ".join(f"{name}: {value}" for name, value in fields if value)
query = "Represent this sentence for searching relevant passages: " + protocol
return model.encode(query, normalize_embeddings=True)
def encode_article(title, abstract):
document = f"Title: {title}. Abstract: {abstract}"
return model.encode(document, normalize_embeddings=True)
The protocol query concatenates the research question, Population, Intervention or Exposure, Comparison, and Outcome. It does not use the source-review title. Documents use title and abstract. Retrieval uses cosine similarity over normalized embeddings.
Training
We used 40 reviews sampled from the training split to select the epoch count. Recall@20 is the primary metric, with Recall@100 and Recall@200 used as tie-breakers. The released checkpoint was retrained for one epoch using the full training split with Multiple Negatives Ranking Loss, batch size 64, learning rate 2e-5, maximum sequence length 512, and seed 718. After excluding articles linked to a test review, 334 of the 336 training reviews contribute positive pairs.
There are 5,585 constructed training pairs. The sentence-transformers training loader used 5,568 examples in complete batches during the epoch; this accounts for the smaller sample count shown by automatically generated trainer metadata.
Test retrieval
All metrics are macro-averaged over the 86 held-out reviews after removing the source review itself from each ranking.
| Metric | K=5 | K=10 | K=20 | K=50 | K=100 | K=200 |
|---|---|---|---|---|---|---|
| Recall@K | 24.0% | 38.8% | 53.5% | 75.3% | 84.2% | 91.7% |
| Precision@K | 47.9% | 42.0% | 34.0% | 22.6% | 14.0% | 8.4% |
The source-review split is disjoint, and every article linked to a test review is excluded as a positive retriever-training example.
Limitations
The model is trained on reviews from Nature Portfolio and a PubMed-centered corpus. Performance may not transfer unchanged to other databases, languages, or domains. A high-recall pool still requires protocol-based screening; model scores should not be treated as inclusion decisions.
Citation
@misc{metasyn2026,
title = {Benchmarking {LLM} Agents on Meta-Analysis Articles from {Nature} Portfolio},
author = {Anzhe Xie and Weihang Su and Yujia Zhou and Yiqun Liu and Min Zhang and Qingyao Ai},
year = {2026},
eprint = {2606.17041},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
url = {https://arxiv.org/abs/2606.17041}
}