PubMed-Ophtha CLIP (ViT-B-16)

Also available: ViT-B-32 · ViT-L-14

Not the paper's checkpoints. This model was trained on a larger split than the models evaluated in the paper (see Model details). The checkpoints behind the paper's reported numbers are in pubmed-ophtha/experiment-checkpoints.

A CLIP model for ophthalmic fundus imaging, finetuned on PubMed-Ophtha — domain-specific scientific literature — which outperforms CLIP models finetuned on text templates, medical reports, and general biomedical literature (BiomedCLIP) across 110 clinical tasks in six fundus imaging datasets (k-NN, zero-shot, and linear probing; see Evaluation scores below). From the paper "Scientific Domain Knowledge Improves Vision-Language Fundus Models" (Hallitschke et al., 2026).

Usage

Zero-shot prediction, using the same prompt style as the paper ("A fundus image showing {-/no} {disease}."):

import open_clip
import torch
from PIL import Image

model, _, preprocess = open_clip.create_model_and_transforms("hf-hub:pubmed-ophtha/PubMed-Ophtha-CLIP-ViT-B-16")
tokenizer = open_clip.get_tokenizer("hf-hub:pubmed-ophtha/PubMed-Ophtha-CLIP-ViT-B-16")
model.eval()

image = preprocess(Image.open("fundus.jpg")).unsqueeze(0)
labels = [
    "a fundus image showing diabetic retinopathy",
    "a fundus image showing no diabetic retinopathy",
]
text = tokenizer(labels)

with torch.no_grad():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features /= text_features.norm(dim=-1, keepdim=True)
    probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)

print(dict(zip(labels, probs[0].tolist())))

Evaluation scores

This is the paper's main/headline architecture: the primary comparison against simplistic templates, augmented templates, medical reports, and a general biomedical literature baseline all use ViT-B/16 as the vision encoder. Those numbers were computed for a model trained on the standard PubMed-Ophtha split rather than this pubmed_ophtha_full checkpoint.

Mean AUROC (%, higher is better) on the same 110-task, 6-dataset benchmark used in the paper (k-NN, zero-shot, and linear-probing protocols), computed directly for this checkpoint (seed 524601411, epoch 50) rather than copied from the paper's own tables, which used a differently split checkpoint (see Model details below). This checkpoint was trained on more data than the paper's models (96,916 vs. 69,373 pairs), which is the likely reason its scores sit slightly above the corresponding numbers in the paper.

DR: diabetic retinopathy, AMD: age-related macular degeneration, HR: hypertensive retinopathy, AH: asteroid hyalosis, MS: macular scars.

Model details

• Vision encoder: ViT-B/16 (patch size 16)
• Image size: 224×224 pixels
• Text encoder: 12-layer transformer (standard OpenCLIP text tower, context length 77)
• Embedding dimension: 512
• Initialized from: OpenAI's CLIP pretrained weights (openai tag)
• Finetuning data: PubMed-Ophtha's pubmed_ophtha_full split (PubMedOphthaFull in pubmed_ophtha_dataset.py). Of PubMed-Ophtha's 102,023 panel/subcaption pairs (all imaging modalities: CFP, OCT, retinal imaging, other), this split stratifies by imaging type and mark status into 96,916 training and 5,102 validation pairs, with no held-out test split, to maximize the data available to this released model. PubMed-Ophtha itself ships no official train/val/test partition (the HF dataset release is a single parquet file); this split is created by this repo's dataset-loading code at finetuning time, and doesn't affect the downstream evaluation benchmark below, which is external to PubMed-Ophtha.
• Training: 50 epochs, batch size 2048, AdamW, cosine learning rate schedule with a 5-epoch linear warmup, single NVIDIA A100 (40GB), via OpenCLIP
• Seed: 524601411
• Activation function: GELU, not QuickGELU — open_clip_config.json omits quick_gelu, matching how this checkpoint was actually finetuned.

Intended use

This is a research resource for contrastive vision-language pretraining/finetuning and retrieval research in ophthalmology. It is not a clinical model, was not validated for diagnostic use, and should not be used for patient care.

Citation

@misc{hallitschke2026scientific,
      title={Scientific Domain Knowledge Improves Vision-Language Fundus Models}, 
      author={Verena Jasmin Hallitschke and Carsten Eickhoff and Philipp Berens},
      year={2026},
      eprint={2605.02720},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2605.02720}, 
}

License

This checkpoint (open_clip_config.json + weights) is released under the MIT license, matching this project's code and the released figure-parsing detection models. It was finetuned starting from OpenAI's CLIP weights on the PubMed-Ophtha dataset; note that PubMed-Ophtha's underlying panels retain the mixed per-article licenses of their source publications — see the dataset's own licensing section for details.

Resources