MedDr: Diagnosis-Guided Bootstrapping for Large-Scale Medical Vision-Language Learning
A generalist foundation model for healthcare capable of handling diverse medical data modalities.
Authors: Sunan He*, Yuxiang Nie*, Zhixuan Chen, Zhiyuan Cai, Hongmei Wang, Shu Yang, Hao Chen**
(*Equal Contribution, **Corresponding Author)
Institution: SMART Lab, Hong Kong University of Science and Technology
Model Summary
MedDr is a large-scale generalist vision-language model for healthcare. It is built upon InternVL and trained using a diagnosis-guided bootstrapping strategy that leverages both image and label information to construct high-quality vision-language datasets.
MedDr supports diverse medical imaging modalities:
- 🫁 Radiology (X-ray, CT, MRI)
- 🔬 Pathology
- 🧴 Dermatology
- 👁️ Retinography
- 🔭 Endoscopy
During inference, MedDr employs a retrieval-augmented medical diagnosis strategy to enhance generalization ability.
Capabilities
- Visual Question Answering (VQA) for medical images
- Medical report generation
- Medical image diagnosis across multiple modalities
Usage
Environment Setup
This model is built on InternVL. Please follow the INSTALLATION.md to set up the environment.
Quick Demo
# Clone the GitHub repository
# git clone https://github.com/sunanhe/MedDr.git
# Edit demo.py and set model_path to your local checkpoint directory
# Then run:
# python3 demo.py
See demo.py in the GitHub repository for a full example.
Citation
If you find MedDr useful in your research, please consider citing:
@article{he2024meddr,
title={MedDr: Diagnosis-Guided Bootstrapping for Large-Scale Medical Vision-Language Learning},
author={He, Sunan and Nie, Yuxiang and Chen, Zhixuan and Cai, Zhiyuan and Wang, Hongmei and Yang, Shu and Chen, Hao},
journal={arXiv preprint arXiv:2404.15127},
year={2024}
}
Acknowledgements
This work builds upon InternVL. We thank the InternVL team for their outstanding contributions to the open-source VLM community.