PubMed-Ophtha CLIP (ViT-B-32)

Also available: ViT-B-16 · ViT-L-14

Not the paper's checkpoints. This model was trained on a larger split than the models evaluated in the paper (see Model details). The checkpoints behind the paper's reported numbers are in pubmed-ophtha/experiment-checkpoints.

A CLIP model for ophthalmic fundus imaging, finetuned on PubMed-Ophtha — domain-specific scientific literature — which outperforms CLIP models finetuned on text templates, medical reports, and general biomedical literature (BiomedCLIP) across 110 clinical tasks in six fundus imaging datasets (k-NN, zero-shot, and linear probing; see Evaluation scores below). From the paper "Scientific Domain Knowledge Improves Vision-Language Fundus Models" (Hallitschke et al., 2026).

Usage

Zero-shot prediction, using the same prompt style as the paper ("A fundus image showing {-/no} {disease}."):

import open_clip
import torch
from PIL import Image

model, _, preprocess = open_clip.create_model_and_transforms("hf-hub:pubmed-ophtha/PubMed-Ophtha-CLIP-ViT-B-32")
tokenizer = open_clip.get_tokenizer("hf-hub:pubmed-ophtha/PubMed-Ophtha-CLIP-ViT-B-32")
model.eval()

image = preprocess(Image.open("fundus.jpg")).unsqueeze(0)
labels = [
    "a fundus image showing diabetic retinopathy",
    "a fundus image showing no diabetic retinopathy",
]
text = tokenizer(labels)

with torch.no_grad():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features /= text_features.norm(dim=-1, keepdim=True)
    probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)

print(dict(zip(labels, probs[0].tolist())))

Evaluation scores

This checkpoint is part of the paper's architecture-generalization ablation: the same finetuning procedure was repeated on ResNet-50, ResNet-50x4, ViT-B/16, ViT-B/32, and ViT-L/14 vision encoders to test whether the benefit of training on PubMed-Ophtha holds regardless of architecture. Models finetuned on PubMed-Ophtha had a significantly higher win rate against every other training data source, pooled across the k-NN, zero-shot, and linear-probing settings and across all five architectures (p < 0.001 for each comparison, two-sided binomial test, n = 550); per-architecture numbers were not reported individually in the paper.

Mean AUROC (%, higher is better) on the same 110-task, 6-dataset benchmark used in the paper (k-NN, zero-shot, and linear-probing protocols), computed directly for this checkpoint (seed 524601411, epoch 50) rather than copied from the paper's own tables, which used a differently split checkpoint (see Model details below). This checkpoint was trained on more data than the paper's models (96,916 vs. 69,373 pairs), which is the likely reason its scores sit slightly above the corresponding numbers in the paper.

DR: diabetic retinopathy, AMD: age-related macular degeneration, HR: hypertensive retinopathy, AH: asteroid hyalosis, MS: macular scars.

Model details

• Vision encoder: ViT-B/32 (patch size 32)
• Image size: 224×224 pixels
• Text encoder: 12-layer transformer (standard OpenCLIP text tower, context length 77)
• Embedding dimension: 512
• Initialized from: OpenAI's CLIP pretrained weights (openai tag)
• Finetuning data: PubMed-Ophtha's pubmed_ophtha_full split (PubMedOphthaFull in pubmed_ophtha_dataset.py). Of PubMed-Ophtha's 102,023 panel/subcaption pairs (all imaging modalities: CFP, OCT, retinal imaging, other), this split stratifies by imaging type and mark status into 96,916 training and 5,102 validation pairs, with no held-out test split, to maximize the data available to this released model. PubMed-Ophtha itself ships no official train/val/test partition (the HF dataset release is a single parquet file); this split is created by this repo's dataset-loading code at finetuning time, and doesn't affect the downstream evaluation benchmark below, which is external to PubMed-Ophtha.
• Training: 50 epochs, batch size 2048, AdamW, cosine learning rate schedule with a 5-epoch linear warmup, single NVIDIA A100 (40GB), via OpenCLIP
• Seed: 524601411
• Activation function: GELU, not QuickGELU — open_clip_config.json omits quick_gelu, matching how this checkpoint was actually finetuned.

Intended use

This is a research resource for contrastive vision-language pretraining/finetuning and retrieval research in ophthalmology. It is not a clinical model, was not validated for diagnostic use, and should not be used for patient care.

Citation

@misc{hallitschke2026scientific,
      title={Scientific Domain Knowledge Improves Vision-Language Fundus Models},
      author={Verena Jasmin Hallitschke and Carsten Eickhoff and Philipp Berens},
      year={2026},
      eprint={2605.02720},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2605.02720},
}

License

This checkpoint (open_clip_config.json + weights) is released under the MIT license, matching this project's code and the released figure-parsing detection models. It was finetuned starting from OpenAI's CLIP weights on the PubMed-Ophtha dataset; note that PubMed-Ophtha's underlying panels retain the mixed per-article licenses of their source publications — see the dataset's own licensing section for details.

Resources