Qwen2.5-VL-3B — Per-Dataset Fine-Tuning Derivatives (LoRA + Full FT)
10 fine-tuned derivatives of Qwen/Qwen2.5-VL-3B-Instruct, one LoRA and one full fine-tune for each of 5 vision-language datasets. Language-model backbone only — the vision tower and multimodal projector are frozen in both variants.
⚠️ Research artifacts: each model is trained on a 3,000-example slice of one dataset for 3 epochs.
Shared training recipe (all 10)
- Base:
Qwen/Qwen2.5-VL-3B-Instruct - Frozen: vision tower + multimodal projector (
freeze_vision_tower: true,freeze_multi_modal_projector: true) — only the LM backbone is trained. - Data:
max_samples = 3000per dataset, 3 epochs,cutoff_len = 2048. - Framework: LLaMA-Factory 0.9.3, transformers 4.52.4.
- LoRA variant: target
q_proj, v_proj; rank 16, alpha 32; lr 2e-4. - Full variant: DeepSpeed ZeRO-3; lr 1e-5;
image_max_pixels = 401408; trainable = 100 % of the LM (vision/projector still frozen).
Contents
| Path | Type | train_loss (3 ep) |
|---|---|---|
lora/mathv360k | LoRA (q/v r16) | 0.160 |
lora/paintingform | LoRA (q/v r16) | 0.823 |
lora/stvqa | LoRA (q/v r16) | 0.437 |
lora/textvqa | LoRA (q/v r16) | 0.212 |
lora/visual7w | LoRA (q/v r16) | 0.545 |
full/mathv360k | Full FT (ZeRO-3) | 0.137 |
full/paintingform | Full FT (ZeRO-3) | 0.659 |
full/stvqa | Full FT (ZeRO-3) | 0.289 |
full/textvqa | Full FT (ZeRO-3) | 0.192 |
full/visual7w | Full FT (ZeRO-3) | 0.386 |
Datasets: MathV360K, PaintingForm, ST-VQA, TextVQA, Visual7W.
Usage
LoRA (base + PEFT):
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from peft import PeftModel
base = Qwen2_5_VLForConditionalGeneration.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct", torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(base, "DSA1/qwen25vl-3b-perdataset-derivatives", subfolder="lora/textvqa")
proc = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct")
Full FT (self-contained LM weights; processor/chat-template from base):
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
model = Qwen2_5_VLForConditionalGeneration.from_pretrained("DSA1/qwen25vl-3b-perdataset-derivatives", subfolder="full/textvqa", torch_dtype="auto", device_map="auto")
proc = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct")
Notes
train_loss= mean training loss over all optimizer steps (1014 steps per model at global batch 8 for LoRA; 129 steps at global batch 64 for full).- Full-FT models ship LM weights only; the vision tower + projector are byte-identical to the base — load processor/chat-template from
Qwen/Qwen2.5-VL-3B-Instruct. - A 7B counterpart of these derivatives exists separately.
License
Inherits the base model's license (Qwen/Qwen2.5-VL-3B-Instruct). Datasets retain their respective licenses.