HPO-PubMedBERT — Structure-Aware Biomedical Embeddings

This is a neuro-symbolic alignment model that fine-tunes PubMedBERT to bridge the semantic gap between Human Phenotype Ontology (HPO) concepts and clinical literature. It was developed as part of the paper "Structure-Aware Contrastive Learning for Biomedical Embeddings: Bridging the Gap between HPO and Clinical Literature" (IJCAI-ECAI 2026).

The model maps biomedical sentences & phenotype descriptions to a 768-dimensional dense vector space optimized for phenotype similarity — two embeddings are close when their associated HPO terms are clinically related (share disease annotations), not merely taxonomically adjacent.

Compared to the base PubMedBERT, this model achieves:

• +9% Spearman ρ on HPO semantic similarity
• +99% Recall@1 on GSC+ gene-disease retrieval
• 4× improvement in Top-1 accuracy on real-world Phenopacket patient retrieval

Model Description

• Base Model: NeuML/pubmedbert-base-embeddings
• Language: English (biomedical domain)
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 768
• Pooling: Mean token embeddings (attention-weighted)
• Similarity Function: Cosine similarity
• Training Data: 270K sentence pairs from PubMed abstracts mentioning HPO terms, supervised by Disease-Overlap (RBP) similarity scores
• Loss Function: AnglE Loss (angle-optimized, avoids gradient saturation)
• Training Strategy: Discriminative layer-wise learning rates, bottom 6 encoder layers frozen

Usage

Sentence-Transformers (recommended)

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("Mellandd/hpo_pubmedbert-rbp-angle")
embeddings = model.encode([
    "Abnormality of the nervous system",
    "Seizures and neurodevelopmental delay"
])

# Compute cosine similarity
from sentence_transformers import util
similarity = util.cos_sim(embeddings[0], embeddings[1])

Hugging Face Transformers

from transformers import AutoTokenizer, AutoModel
import torch

def mean_pooling(output, mask):
    embeddings = output[0]
    mask = mask.unsqueeze(-1).expand(embeddings.size()).float()
    return torch.sum(embeddings * mask, 1) / torch.clamp(mask.sum(1), min=1e-9)

tokenizer = AutoTokenizer.from_pretrained("Mellandd/hpo_pubmedbert-rbp-angle")
model = AutoModel.from_pretrained("Mellandd/hpo_pubmedbert-rbp-angle")

sentences = ["Abnormality of the nervous system", "Seizures and neurodevelopmental delay"]
inputs = tokenizer(sentences, padding=True, truncation=True, return_tensors="pt")

with torch.no_grad():
    output = model(**inputs)

embeddings = mean_pooling(output, inputs["attention_mask"])

Evaluation Results

HPO Semantic Textual Similarity (STS)

Pearson and Spearman correlation between model cosine similarity and ground-truth Disease-Overlap (RBP) scores on held-out HPO term pairs:

GSC+ — Mention-to-HPO Linking (228 annotated abstracts, 1,933 annotations)

Real-World Phenopacket Patient Retrieval (6,556 clinical cases)

Matching patients by embedding their phenotype profiles:

Training

Dataset

Sentence pairs were generated from PubMed abstracts mentioning Human Phenotype Ontology (HPO) terms, with quality filtering including negation detection, enumeration removal, and dynamic context windows (±25 words). Training pairs were formed via Anchor-Based Hard Sampling:

• 33% Positive: different sentences for the same phenotype
• 33% Hard Negative: terms with moderate RBP similarity (0.3–0.7) — siblings/cousins sharing some diseases
• 33% Random Negative: low-similarity terms for global structure preservation

Ground-truth similarity scores use the Disease-Overlap (Relative Best Pair) metric, which measures shared disease annotations between phenotype terms — capturing clinical co-occurrence rather than mere taxonomic proximity.

Hyperparameters

Discriminative Layer-wise Learning Rates

Bottom 6 layers frozen, top 6 unfrozen with linearly increasing learning rates:

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_mean_tokens': True})
)

Intended Use

This model is designed for biomedical phenotype representation and retrieval tasks:

• Semantic similarity between phenotype descriptions
• Patient-to-disease matching (embedding disease phenotype profiles and querying with patient phenotypes)
• Mention-to-HPO concept normalization
• Document-level phenotype indexing and retrieval

It is not intended for general-domain sentence similarity. The model specializes in clinical/biomedical phenotype vocabulary from the HPO.

Limitations and Biases

• Domain-specific: Trained exclusively on PubMed biomedical literature and HPO terminology. Performance degrades on general-domain text.
• Language: English only.
• HPO coverage: Performance correlates with the number of training sentences available per HPO term; rare phenotypes with limited literature mentions may have weaker representations.
• Sequence length: Truncated at 256 tokens, suitable for sentences and short paragraphs but not full-length articles.

Citation

TBD - Will update when the IJCAI-ECAI 2026 proceedings are online. Preprint: Here

Dependencies

• sentence-transformers ≥ 5.1.0
• transformers ≥ 4.57.0
• PyTorch ≥ 2.0

Environment Versions