HealthGPT-Pro-4B

🧭 0. Overview

HealthGPT-Pro is a SoTA medical multimodal large language model (Med-MLLM) built on Qwen3-VL. It is designed for medical text, 2D medical image, and 3D medical volumes understanding and analysis, providing strong performance across broad medical text-based and vision-language tasks.

✨ Core features:

• Multimodal input support: HealthGPT-Pro can process text, 2D images, and 3D volumetric data.
• Efficient training: HealthGPT-Pro achieves SoTA performance through a two-stage training recipe, using 3M samples for alignment and 10M samples for SFT.
• Instruction following: Unlike many medical Med-MLLMs tuned only on medical-domain data, HealthGPT-Pro preserves a substantial proportion of general data to maintain instruction-following ability.
• Comprehensive modality coverage: (1) Computed Tomography (2) Digital Photography (3) Fundus Photography (4) Infrared Reflectance Imaging (5) Magnetic Resonance Imaging (6) Optical Coherence Tomography (7) Dermoscopy (8) Endoscopy (9) Microscopy (10) X-ray Imaging (11) Ultrasound Imaging (12) Histopathology (13) Colposcopy (14) Text.
• Comprehensive task coverage: HealthGPT-Pro is trained on diverse medical and general tasks, with strong instruction-following behavior.

This model is intended for research use. It should not be used as a substitute for professional clinical judgment, diagnosis, or treatment.

📊 1. Performance Comparison

📝 Medical Text Benchmarks

🖼️ Medical Multimodal Benchmarks

⚙️ 2. Environment Setup

The recommended environment is provided as follows:

# Create and activate a clean Python 3.12 environment
conda create -n healthgpt-pro python=3.12 -y
conda activate healthgpt-pro

# Install PyTorch with CUDA support
# If your CUDA version is lower than 12.8, install a matching PyTorch build instead (e.g., cu121 or cu118).
pip install torch==2.8.0 torchvision==0.23.0 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu128

# Install FlashAttention for faster attention
pip install flash-attn==2.8.3 --no-build-isolation --upgrade

# Install other dependencies
pip install transformers==4.57.1 accelerate==1.11.0 deepspeed==0.16.9 numpy==1.26.4 peft==0.17.1
pip install qwen-vl-utils pillow

🚀 3. Inference

🧩 Load Model and Processor

import numpy as np
import torch
from PIL import Image
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration
from qwen_vl_utils import process_vision_info

model_id = "HealthGPT-Pro-4B"

model = Qwen3VLForConditionalGeneration.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(model_id)

💬 Text-Only Inference

messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "Explain the key symptoms and common risk factors of pneumonia."},
        ],
    }
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

generated_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed,
    skip_special_tokens=True,
    clean_up_tokenization_spaces=False,
)
print(output_text[0])

🩻 Single-Image Inference

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "examples/chest_xray.png"},
            {"type": "text", "text": "Describe the main radiological findings in this image."},
        ],
    }
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

generated_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed,
    skip_special_tokens=True,
    clean_up_tokenization_spaces=False,
)
print(output_text[0])

🖼️ Multi-Image Inference

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "examples/image_1.png"},
            {"type": "image", "image": "examples/image_2.png"},
            {"type": "text", "text": "Compare these two medical images and summarize the key differences."},
        ],
    }
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

generated_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed,
    skip_special_tokens=True,
    clean_up_tokenization_spaces=False,
)
print(output_text[0])

🧠 3D Volume Inference

The repository inference path converts a 3D .npy volume into a sequence of 2D frames and sends it as a video-style input.

def ct_to_video(ct_path: str):
    ct_pixels = np.load(ct_path)
    ct_u8 = np.clip(ct_pixels * 255, 0, 255).astype(np.uint8)

    frames = []
    idx = np.linspace(1, len(ct_u8) - 2, 10, dtype=int)
    for i in idx:
        rgb = np.stack([ct_u8[i]] * 3, axis=-1)
        frames.append(Image.fromarray(rgb, mode="RGB"))
    return frames

volume_frames = ct_to_video("examples/ct_volume.npy")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "video", "video": volume_frames, "sample_fps": 2.0},
            {"type": "text", "text": "Analyze this CT volume and summarize the main findings."},
        ],
    }
]

text = processor.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
images, videos, video_kwargs = process_vision_info(
    messages,
    image_patch_size=16,
    return_video_kwargs=True,
    return_video_metadata=True,
)
if videos is not None:
    videos, video_metadatas = zip(*videos)
    videos, video_metadatas = list(videos), list(video_metadatas)
else:
    video_metadatas = None

inputs = processor(
    text=text,
    images=images,
    videos=videos,
    video_metadata=video_metadatas,
    return_tensors="pt",
    do_resize=False,
    **video_kwargs,
).to(model.device)

generated_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed,
    skip_special_tokens=True,
    clean_up_tokenization_spaces=False,
)
print(output_text[0])

📚 4. Citation

If you find this model useful for your research, please cite:

@misc{lin2025healthgptmedicallargevisionlanguage,
      title={HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation}, 
      author={Tianwei Lin and Wenqiao Zhang and Sijing Li and Yuqian Yuan and Binhe Yu and Haoyuan Li and Wanggui He and Hao Jiang and Mengze Li and Xiaohui Song and Siliang Tang and Jun Xiao and Hui Lin and Yueting Zhuang and Beng Chin Ooi},
      year={2025},
      eprint={2502.09838},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2502.09838}, 
}