Qwen3-VL-2B Cataract Surgery: SFT Stage 2 (Continued Fine-Tuning)
This repository contains the standalone, fully merged 16-bit model weights for Stage 2 Supervised Fine-Tuning (SFT) of Qwen3-VL-2B on cataract surgical video narration and visual understanding.
Stage 2 continued fine-tuning specifically refines long-horizon temporal narrative coherence across whole surgical cases by concentrating training on all full cataract surgery videos while anchoring clip-level representations with stratified surgical clips.
🔬 SFT Stage 2 Training Overview
- Base Model Initialization:
shahedm2001/qwen3-vl-2b-cataract-sft(Stage 1 merged checkpoint). - Training Data Composition:
- 100% of Full Videos: All 108 unclipped surgery recordings (capturing complete procedural arcs from corneal incision to wound closure).
- 10% Stratified Clips: Sampled equally across YouTube cataract surgical demonstrations and clinical Phase recordings from
shahedm2001/dataset_sft.
- Curriculum Design & Adaptations:
- Halved Learning Rates: LLM LoRA rate was halved to ( 5 \times 10^{-5} ), merger LR to ( 5 \times 10^{-6} ), and vision transformer LR to ( 1 \times 10^{-6} ) to stabilize learned visual features while adapting to extended temporal narratives.
- Rank 8 / Alpha 16 LoRA Parameterization: Targeted adapter capacity preserving base surgical representations.
- 64 Frames at Native Aspect Ratios: Dynamic visual token budget (98,304 to 196,608 pixels per frame) capturing fine-grained instrument movements and ocular tissue boundaries.
📈 Stage 2 Training Dynamics & Convergence
During Stage 2 fine-tuning, cross-entropy token loss dropped from 0.9812 down to a new global minimum of 0.4132, accompanied by a 2.3× stabilization in gradient norms compared to Stage 1.
Stage 2 Standalone Dynamics (Steps 1–42)
SFT Two-Stage Concatenated Progression (Steps 1–742)
📊 Quantitative Training Metrics
| Telemetry / Hyperparameter | Stage 1 (Full SFT) | Stage 2 (Continued SFT) | Cumulative / Final SFT |
|---|---|---|---|
| Base Model | Qwen/Qwen3-VL-2B-Instruct | shahedm2001/qwen3-vl-2b-cataract-sft | Base Model |
| Data Split | Full dataset_sft (5,597 items) | All 108 Full Videos + 10% Clips | 5,597 unique videos |
| LoRA Parameterization | Rank 16, Alpha 32 | Rank 8, Alpha 16 | Rank 8–16 Adaptations |
| Peak LLM Learning Rate | ( 1.0 \times 10^{-4} ) (cosine) | ( 5.0 \times 10^{-5} ) (cosine, halved) | Warmup + Cosine |
| Optimizer Steps | 700 steps (2.0 epochs) | 42 steps (1.0 epoch) | 742 global steps |
| Initial Training Loss | 2.2810 | 0.9812 | 2.2810 |
| Minimum Training Loss | 0.6804 | 0.4132 | 0.4132 |
| Final Step Loss | 1.0620 | 0.4132 | 0.4132 |
| Median Gradient Norm | 2.56 | 1.09 (2.3× lower) | 2.45 |
| Total Wall-Time | ~11.8 hours | 41.7 minutes | ~12.5 hours |
📦 Training Logs & Telemetry Artifacts
The complete logs, console outputs, and hardware metrics for Stage 2 (and the concatenated 742-step telemetry) are included in this repository:
training_logs/logs.zip— Raw training logs and telemetry per sub-epoch.training_logs/sft_training_logs_stages_1_and_2.zip— Complete bundled archive containing rawtrain.log,losses.csv,gpu.csv,cmd.txt, andconfig.txtacross both stages.plots/figure_sft_stage2_dynamics.png— Dedicated 4-panel Stage 2 training dashboard.plots/figure_sft_two_stage_training_dynamics.png— Unified 742-step concatenated training dynamics.
💻 Usage & Inference
The model is a standalone, merged 16-bit model compatible with standard Hugging Face transformers:
import torch
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration
from qwen_vl_utils import process_vision_info
model_id = "shahedm2001/qwen3-vl-2b-cataract-sft-stage2"
# 1. Load processor and standalone model
processor = AutoProcessor.from_pretrained(model_id)
model = Qwen3VLForConditionalGeneration.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="cuda:0"
)
# 2. Prepare surgical video query
messages = [
{
"role": "user",
"content": [
{"type": "video", "video": "path/to/cataract_video.mp4", "nframes": 64},
{"type": "text", "text": "Describe the anatomical surgical steps and instrument interactions visible in this cataract operation clip."}
]
}
]
# 3. Process video and generate narration
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
text=[text],
images=image_inputs,
videos=video_inputs,
padding=True,
return_tensors="pt"
).to("cuda:0")
with torch.no_grad():
output_ids = model.generate(**inputs, max_new_tokens=256, do_sample=False)
trimmed_ids = output_ids[0][len(inputs.input_ids[0]):]
response = processor.decode(trimmed_ids, skip_special_tokens=True).strip()
print(response)
🔗 Related Resources
- Stage 1 SFT Model:
shahedm2001/qwen3-vl-2b-cataract-sft - Final GRPO RL Model:
shahedm2001/qwen3-vl-2b-cataract-grpo - Dataset Repository:
shahedm2001/dataset_sft - GitHub Codebase:
https://github.com/shahedmomenzadeh/qwen3-VL-2B-finetune