DermoGPT-RL
DermoGPT-RL is a dermatology-oriented multimodal model for research on dermatology image understanding, lesion description, morphology-aware reasoning, and medical visual question answering.
This repository contains a merged Hugging Face Transformers checkpoint based on a Qwen3-VL architecture and further optimized for dermatology-oriented instruction following and reasoning.
Model Details
- Model type: vision-language model
- Architecture:
Qwen3VLForConditionalGeneration - Precision: float16
- Format: sharded
safetensors - Number of shards: 4
- Repository size: approximately 17 GB
- Primary language: English
- Domain: dermatology and dermoscopy
Intended Use
DermoGPT-RL is intended for:
- research on dermatology-oriented multimodal models;
- medical visual question answering under controlled research settings;
- lesion morphology description and reasoning;
- benchmark evaluation on DermoGPT/DermoBench-style tasks;
- further academic analysis of reinforcement learning for medical VLMs.
Out-of-Scope Use
This model is not intended for:
- clinical deployment;
- autonomous diagnosis;
- patient triage;
- treatment recommendation;
- direct patient-facing medical advice;
- replacement of professional medical judgment.
Loading
Install recent versions of the required libraries:
pip install -U transformers accelerate safetensors pillow
Example loading code:
import torch
from PIL import Image
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration
model_id = "mendicant04/DermoGPT-RL"
model = Qwen3VLForConditionalGeneration.from_pretrained(
model_id,
torch_dtype=torch.float16,
device_map="auto",
)
processor = AutoProcessor.from_pretrained(model_id)
image = Image.open("path/to/dermatology_image.jpg").convert("RGB")
messages = [
{
"role": "user",
"content": [
{"type": "image", "image": image},
{"type": "text", "text": "Describe the lesion and summarize the key morphological findings."},
],
}
]
text = processor.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
)
inputs = processor(
text=[text],
images=[image],
return_tensors="pt",
).to(model.device)
with torch.no_grad():
generated_ids = model.generate(
**inputs,
max_new_tokens=512,
do_sample=False,
)
output = processor.batch_decode(
generated_ids,
skip_special_tokens=True,
clean_up_tokenization_spaces=False,
)[0]
print(output)
Related Resources
- Project repository: https://github.com/mendicant04/DermoGPT
- Benchmark/data resources: https://huggingface.co/mendicant04
- Codebase adapted from: https://github.com/2U1/Qwen-VL-Series-Finetune
Limitations
- The model may produce incorrect, incomplete, or overconfident outputs.
- Performance may vary across skin tones, imaging devices, disease categories, and acquisition conditions.
- Dermatology reasoning outputs should be interpreted as research model outputs, not as clinical advice.
- Any downstream use requires independent validation and appropriate medical oversight.
Ethical and Medical Considerations
Dermatology is a high-stakes medical domain. This model should only be used for research and evaluation. It must not be used as a standalone diagnostic or treatment system. Researchers are responsible for ensuring that their use complies with applicable laws, institutional review requirements, data-use agreements, and medical AI safety standards.