DermLIP: Dermatology Language-Image Pretraining

Model Description

DermLIP is a vision-language model for dermatology, trained on the Derm1M dataset—the largest dermatological image-text corpus to date.

Model Details

• Model Type: Pretrained Vision-Language Model (CLIP-style)

• Architecture:

  • Vision encoder: ViT-B16
  • Text encoder: GPT2

• Resolution: 224×224 pixels

• Paper: https://arxiv.org/abs/2503.14911

• Repository: https://github.com/SiyuanYan1/Derm1M

• license: cc-by-nc-nd-4.0

Training Details

• Training data: 403,563 skin image-text pairs from Derm1M datasets. Images include both dermoscopic and clinical images.
• Training objective: image-text contrastive loss
• Hardware: 1 x Nvidia H200 （～40GB memory usage）
• Hours used: ~5 hours

Intended Uses

Primary Use Cases

• Zero-shot classification
• Few-shot learning
• Cross-modal retrieval
• Concept annotation/explanation

How to Use

Installation

Accept the access terms above and log in once with huggingface-cli login. Then clone the Derm1M repository:

git clone git@github.com:SiyuanYan1/Derm1M.git
cd Derm1M
···

Then install the package following the instruction in the repository.


### Quick Start
```python
import open_clip
from PIL import Image
import torch

# Load model with huggingface checkpoint
model, _, preprocess = open_clip.create_model_and_transforms(
    'hf-hub:redlessone/DermLIP_ViT-B-16'
)
model.eval()

# Initialize tokenizer
tokenizer = open_clip.get_tokenizer('hf-hub:redlessone/DermLIP_ViT-B-16')

# Read example image
image = preprocess(Image.open("your_skin_image.png")).unsqueeze(0)

# Define disease labels (example: PAD dataset classes)
PAD_CLASSNAMES = [
    "nevus",
    "basal cell carcinoma",
    "actinic keratosis",
    "seborrheic keratosis",
    "squamous cell carcinoma",
    "melanoma"
]

# Build text prompts
template = lambda c: f'This is a skin image of {c}'
text = tokenizer([template(c) for c in PAD_CLASSNAMES])

# Inference
with torch.no_grad(), torch.autocast("cuda"):
    # Encode image and text
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    
    # Normalize features
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features /= text_features.norm(dim=-1, keepdim=True)
    
    # Compute similarity
    text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)

# Get prediction
final_prediction = PAD_CLASSNAMES[torch.argmax(text_probs[0])]
print(f'This image is diagnosed as {final_prediction}.')
print("Label probabilities:", text_probs)

Contact

For any additional questions or comments, contact Siyuan Yan (siyuan.yan@monash.edu),

Cite our Paper

@misc{yan2025derm1m,
  title        = {Derm1M: A Million‑Scale Vision‑Language Dataset Aligned with Clinical Ontology Knowledge for Dermatology},
  author       = {Siyuan Yan and Ming Hu and Yiwen Jiang and Xieji Li and Hao Fei and Philipp Tschandl and Harald Kittler and Zongyuan Ge},
  year         = {2025},
  eprint       = {2503.14911},
  archivePrefix= {arXiv},
  primaryClass = {cs.CV},
  url          = {https://arxiv.org/abs/2503.14911}
}

@article{yan2025multimodal,
  title={A multimodal vision foundation model for clinical dermatology},
  author={Yan, Siyuan and Yu, Zhen and Primiero, Clare and Vico-Alonso, Cristina and Wang, Zhonghua and Yang, Litao and Tschandl, Philipp and Hu, Ming and Ju, Lie and Tan, Gin and others},
  journal={Nature Medicine},
  pages={1--12},
  year={2025},
  publisher={Nature Publishing Group}
}