SIREN Screening Bi-encoder

A bi-encoder model for systematic review screening, trained to retrieve documents matching eligibility criteria. Achieves +24 percentage points MRR@10 over MedCPT on criteria-based retrieval.

Model Details

Intended Use

Primary use case: First-stage retrieval for systematic review screening pipelines.

Given eligibility criteria (e.g., "RCTs of aspirin in adults with diabetes, published after 2015"), retrieve candidate documents from a corpus for human review or downstream reranking.

Recommended pipeline:

1. Encode eligibility criteria with this bi-encoder
2. Retrieve top-k candidates via similarity search
3. Rerank with siren-screening-crossencoder for 3-class classification

Usage

Sentence-Transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("Praise2112/siren-screening-biencoder")

# Encode eligibility criteria (query)
query = "Randomized controlled trials of aspirin for cardiovascular prevention in diabetic adults"
query_embedding = model.encode(query)

# Encode candidate documents
documents = [
    "A randomized trial of low-dose aspirin in 5,000 diabetic patients showed reduced MI risk...",
    "This cohort study examined statin use in elderly populations with hyperlipidemia...",
]
doc_embeddings = model.encode(documents)

# Compute similarity
similarities = model.similarity(query_embedding, doc_embeddings)
print(similarities)  # tensor([[0.85, 0.42]])

Transformers

import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("Praise2112/siren-screening-biencoder")
model = AutoModel.from_pretrained("Praise2112/siren-screening-biencoder")

def encode(texts):
    inputs = tokenizer(texts, padding=True, truncation=True, max_length=768, return_tensors="pt")
    with torch.no_grad():
        outputs = model(**inputs)
    # Mean pooling
    attention_mask = inputs["attention_mask"]
    embeddings = outputs.last_hidden_state
    mask_expanded = attention_mask.unsqueeze(-1).expand(embeddings.size()).float()
    sum_embeddings = torch.sum(embeddings * mask_expanded, dim=1)
    sum_mask = torch.clamp(mask_expanded.sum(dim=1), min=1e-9)
    return F.normalize(sum_embeddings / sum_mask, p=2, dim=1)

query_emb = encode(["RCTs of aspirin in diabetic patients"])
doc_emb = encode(["A randomized trial of aspirin in 5,000 diabetic patients..."])
similarity = torch.mm(query_emb, doc_emb.T)

Performance

Internal Benchmark (SIREN Test Set)

This is +24 percentage points MRR@10 over MedCPT, the leading biomedical retrieval model.

External Benchmarks

Training

This model was created by:

1. Fine-tuning gte-modernbert-base on the siren-screening dataset
2. SLERP merging with the base model (t=0.4) to preserve out-of-distribution generalization

Training details:

• Loss: Multiple Negatives Ranking Loss (MNRL) with in-batch negatives
• Batch size: 512 (via GradCache)
• Hard negatives: BM25-mined + LLM-generated
• Matryoshka dimensions: [768, 512, 256, 128, 64]
• Epochs: 1

Limitations

• Synthetic queries, real documents: The queries and relevance labels are LLM-generated, but the documents are real PubMed articles
• English only: Trained on English PubMed articles
• Not a classifier: This model retrieves candidates; use the cross-encoder for relevance classification

Citation

@misc{oketola2026siren,
  title={SIREN: Improving Systematic Review Screening with Synthetic Training Data for Neural Retrievers},
  author={Praise Oketola},
  year={2026},
  howpublished={\url{https://huggingface.co/Praise2112/siren-screening-biencoder}},
  note={Bi-encoder model}
}

License

Apache 2.0