Oculora

A unified ophthalmic VLM for understanding and clinical generation

1.33 M image–text pairs · 360 K ophthalmic instructions · 7 imaging modalities · 300+ ophthalmic conditions

⚠️ Research use only. Oculora is a research artifact — not a medical device. It is not cleared or approved by any regulatory body and must not be used for diagnosis, triage or treatment decisions for real patients. See Intended use and limitations.

Highlights

• Ophthalmology specialist — Qwen2.5-VL-7B adapted through three stages: pretraining on 1.33 M image–text pairs, supervised fine-tuning on 360 K ophthalmic instructions, and risk-aware DAPO policy optimization.
• Two evaluation axes — multimodal understanding (multiple-choice and short-answer questions) and open-ended clinical generation (report generation and medical consultation).
• Broad coverage — a 22,000-sample internal test set plus four external benchmarks spanning seven imaging modalities and over 300 ophthalmic conditions.
• Leads every free-text benchmark — first on every generation metric across OphthalVQA, FFA-IR and JAMA Challenge against Qwen3-VL, MedGemma and Lingshu (all P < 0.001).
• Clinically reviewed — ranked highest overall for correctness and control of unsupported content in a 50-case blinded review, with 0.94 agreement (95% CI 0.90–0.97) between LLM-judge and ophthalmologist rankings.
• Reproducible out of the box — the shipped generation_config.json and preprocessor_config.json reproduce the evaluated configuration exactly.

Previous ophthalmic vision–language model (VLM) studies have commonly evaluated visual question answering and report generation, with limited evidence for medical consultation. Medical consultation requires models to synthesize clinical context and imaging findings into diagnostic and management-oriented responses without predefined answers or explicit diagnostic cues. Here we developed Oculora, a 7B-scale ophthalmic VLM trained on 1.33 million image–text pairs and 360,000 ophthalmic instructions through staged pretraining, supervised fine-tuning and risk-aware policy optimization. We assessed multimodal understanding with multiple-choice and short-answer questions and open-ended clinical generation with report generation and medical consultation on a 22,000-sample internal test set and four benchmarks spanning seven modalities and over 300 ophthalmic conditions. Evaluation combined prespecified task-specific metrics with five-point clinical-quality ratings from large language model judges and senior ophthalmologists. Multiple-choice accuracy was 0.820 internally and 0.674 on JSIEC. Oculora led all generation metrics across three external free-text benchmarks (all P < 0.001). On JAMA Challenge, ROUGE-L was 0.500 versus 0.311 and BERTScore-F1 was 0.927 versus 0.893 for Qwen3-VL-8B. In a 50-case blinded review, Oculora ranked highest overall for correctness and control of unsupported content; LLM and ophthalmologist rankings agreed (0.94, 95% CI 0.90–0.97). These findings show that a specialized ophthalmic VLM can support both multimodal understanding and open-ended clinical generation.

Quickstart

Requires transformers >= 4.57.1. Greedy decoding and max_pixels = 262144 are already the shipped defaults, so loading the model as-is reproduces the reported configuration — no extra flags required.

import torch
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor

model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "pxiaoyu/Oculora",
    dtype=torch.bfloat16,
    device_map="auto",
)

# max_pixels = 262144 is already the shipped default
processor = AutoProcessor.from_pretrained("pxiaoyu/Oculora")

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "url": "https://example.com/fundus.jpg"},
        {"type": "text", "text": "Describe the fundus findings and give the most likely diagnosis."},
    ],
}]

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

# greedy decoding is already the shipped default
out = model.generate(**inputs, max_new_tokens=1000)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

python -m vllm.entrypoints.openai.api_server \
  --model pxiaoyu/Oculora \
  --served-model-name oculora \
  --host 127.0.0.1 --port 8002 \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.75 \
  --max-num-seqs 8 \
  --limit-mm-per-prompt '{"image":4}' \
  --trust-remote-code

Query it as any chat-completions endpoint with image_url content parts. vLLM reads generation_config.json from the repo, so "temperature": 0 is already the default.

Hardware — bfloat16 weights are ~17 GB plus KV cache; a single 40 GB A100 or 48 GB L40S is comfortable at --max-model-len 8192.

Evaluation

Multiple-choice and short-answer questions assess multimodal understanding; report generation and medical consultation assess open-ended clinical generation. Every model below was evaluated under an identical protocol on the same test sets.

Best score per row in bold. Test sets: JSIEC n = 1,000 · OphthalVQA n = 600 · FFA-IR n = 766 · JAMA Challenge n = 177, plus a 22,000-sample internal test set. Baselines are open-weight vision–language models at a comparable parameter scale, run under an identical protocol on the same test sets: Qwen3-VL-8B, MedGemma-4B and Lingshu-7B. Bootstrap 95% confidence intervals for every value are reported in the paper.

Blinded clinical review

Training

Three stages, all on the Qwen2.5-VL-7B backbone.

Risk-aware DAPO

Masking the riskiest decile globally rather than within each group is what stabilizes the gradients: the advantage baseline is never computed over generations the risk model has already flagged.

Inference settings used for all reported results

These are the shipped defaults. preprocessor_config.json sets max_pixels = 262144 and generation_config.json sets do_sample = false, so loading the model as-is reproduces the reported configuration — no extra flags required.

Architecture

Intended use and limitations

Oculora is a research artifact, intended for research on ophthalmic vision–language modelling and for retrospective evaluation on de-identified data.

• Not a medical device. Not cleared or approved by any regulatory body. It must not be used for diagnosis, triage or treatment decisions for real patients.
• Outputs can be fluent and confident while being wrong. Every clinical statement requires review by a qualified ophthalmologist.
• Performance is characterised on the benchmarks above — seven imaging modalities and 300+ conditions. Behaviour outside that distribution (other modalities, populations, languages, or image quality) is unmeasured.
• Training data inherits the demographic and disease-prevalence biases of its sources; the model may underperform on rare conditions and underrepresented groups.
• Do not send protected health information to any hosted inference endpoint.

License

Apache 2.0, inheriting the license of the Qwen2.5-VL base model. Users remain responsible for complying with the terms of the underlying base model and of every dataset used for evaluation.

Citation

The paper describing Oculora is currently under review; a full citation will be added here upon publication. Until then, please cite this repository: pxiaoyu/Oculora.