MAGEN-O-MAKE (ViT-B/16)

Vision-language model for dermatology, pretrained with MAGEN (Multi-Agent data GENeration) and O-MAKE (Ontology-based Multi-Aspect Knowledge-Enhanced pretraining).

MAGEN synthesizes knowledge-enriched image descriptions through a foundation-model-assisted captioning and retrieval-based verification pipeline. O-MAKE decomposes those long clinical texts into distinct knowledge aspects and aligns them with image features at both the global and patch level, guided by a dermatological ontology.

• 📄 Paper: IEEE TMI (arXiv:2512.03445)
• 💻 Code: github.com/XiejiLi/MAGEN-O-MAKE

Model details

Usage

import torch
from PIL import Image
import open_clip

model, preprocess = open_clip.create_model_from_pretrained('hf-hub:Xieji-Li/MAGEN-O-MAKE')
tokenizer = open_clip.get_tokenizer('hf-hub:Xieji-Li/MAGEN-O-MAKE')
model.eval()

# The six PAD-UFES-20 classes.
labels = ['nevus', 'basal cell carcinoma', 'actinic keratosis',
          'seborrheic keratosis', 'squamous cell carcinoma', 'melanoma']

# A biopsy-confirmed melanoma from the PAD-UFES-20 test split.
image = preprocess(Image.open('PAT_611_1158_156.png').convert('RGB')).unsqueeze(0)
text = tokenizer([f'This is a skin image of {c}' for c in labels])

with torch.no_grad():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features /= text_features.norm(dim=-1, keepdim=True)
    probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)[0]

for label, p in sorted(zip(labels, probs.tolist()), key=lambda kv: -kv[1]):
    print(f'{label:25s} {p:.4f}')

melanoma                  0.9227
seborrheic keratosis      0.0682
nevus                     0.0063
basal cell carcinoma      0.0011
actinic keratosis         0.0010
squamous cell carcinoma   0.0006

The image is from PAD-UFES-20; the code repository links a bundle of all downstream datasets. This single-template prompt is the simplest possible use; the benchmarks below ensemble 8 templates (OPENAI_SKIN_TEMPLATES in the code repository).

Zero-shot results

Measured with this checkpoint using the evaluation pipeline in the code repository (bash script/zeroshot_eval.sh), 8-template prompt ensemble:

The last two rows are long-tail splits: rare conditions the model never saw a label for during pretraining.

Files

Intended use and limitations

This model is a research artifact, released for non-commercial research use only (CC BY-NC-ND 4.0). It is not a medical device and must not be used for diagnosis, triage, or any other clinical decision-making.

Known limitations:

• Trained on web-sourced dermatological images; skin-tone, geographic, and condition coverage is uneven, and performance varies accordingly across subgroups.
• Zero-shot accuracy on fine-grained benchmarks with 100+ classes is well below clinical reliability.
• Predictions depend on the prompt wording and on the candidate label set supplied by the caller.
• Part of the pretraining captions are model-generated (MAGEN) and were verified by retrieval rather than by clinicians, so residual label noise is expected.

Citation

@article{li2025multi,
  title={Multi-Aspect Knowledge-Enhanced Medical Vision-Language Pretraining with Multi-Agent Data Generation},
  author={Li, Xieji and Yan, Siyuan and Liu, Yingsheng and Soyer, H Peter and Janda, Monika and Mar, Victoria and Ge, Zongyuan},
  journal={arXiv preprint arXiv:2512.03445},
  year={2025}
}

The MICCAI'25 conference version this extends:

@misc{yan2025makemultiaspectknowledgeenhancedvisionlanguage,
      title={MAKE: Multi-Aspect Knowledge-Enhanced Vision-Language Pretraining for Zero-shot Dermatological Assessment}, 
      author={Siyuan Yan and Xieji Li and Ming Hu and Yiwen Jiang and Zhen Yu and Zongyuan Ge},
      year={2025},
      eprint={2505.09372},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2505.09372}, 
}