Qwen2.5-VL-3B finetuned on MLLMU-Bench (all 500 profiles)
The starting model for unlearning experiments: Qwen/Qwen2.5-VL-3B-Instruct fully finetuned on every fictitious
profile in MLLMU-Bench, so it has learned the facts that the unlearning methods then try to remove. The unlearned models
graddiff-forget5,
npo-forget5,
klmin-forget5 and
manu-forget5 start from this model, and
retain95 is its retain-only counterpart.
Data
MLLMU-Bench contains 500 fictitious people. Each has a face image and
15-20 image-grounded QA pairs plus one biography answer (ft_Data). The benchmark splits the profiles into
forget_5 (25 profiles) and retain_95 (475 profiles). Retain_Set holds 153 real celebrities and is used only
to measure general knowledge (utility).
Training
| Hyperparameter | Value |
|---|---|
| Base model | Qwen/Qwen2.5-VL-3B-Instruct |
| Training data | ft_Data QA pairs of all 500 profiles (8,204 examples: image + question -> answer) |
| Trainable parameters | Full finetune: language model, vision tower (ViT) and merger |
| Loss | Cross-entropy on answer tokens only (prompt/image tokens masked) |
| Optimizer | AdamW, weight decay 0.0 |
| Learning rate | 2e-5, cosine schedule, 3% linear warmup |
| Epochs | 10 (5,130 steps) |
| Batch size | 16 (single GPU, no gradient accumulation) |
| Grad clip | 1.0 (max norm) |
| Precision | bf16, SDPA attention, gradient checkpointing |
| Image resolution | min 128x28x28, max 256x28x28 pixels |
| Seed | 42 |
| Hardware | 1x NVIDIA B200 |
Evaluation
- Task: MLLMU-Bench
Generation_Task, 4 open-ended questions per profile (2Image_Textual+ 2Pure_Text). Every question is asked together with the profile image. Split sizes: forget_5 = 100 questions, retain_95 = 1900, Retain_Set = 612. - Decoding: vLLM, greedy, max 128 new tokens, default Qwen system prompt, images resized to 128-256 x 28x28 pixels.
- ROUGE:
rouge_scorewith stemming, generation vs ground truth (ROUGE-L F1 reported). - LLM judge:
Qwen/Qwen2.5-7B-Instruct, binary per answer: 1 = the answer contains the ground-truth fact or a semantically equivalent one, 0 = wrong, missing, partial or merely similar (e.g. a different city or salary). Judge accuracy is the mean over questions.
Goal of unlearning: forget_5 ↓ (toward the retain-only model), with retain_95 and Retain_Set close to the finetuned model.
| Model | forget_5 ROUGE-L | forget_5 judge ↓ | retain_95 ROUGE-L | retain_95 judge ↑ | Retain_Set ROUGE-L | Retain_Set judge ↑ |
|---|---|---|---|---|---|---|
| Qwen2.5-VL-3B-Instruct (base) | 0.353 | 0.030 | 0.363 | 0.021 | 0.411 | 0.306 |
| qwen2.5-vl-3b-mllmu-ft | 0.699 | 0.490 | 0.727 | 0.634 | 0.487 | 0.199 |
| qwen2.5-vl-3b-mllmu-retain95 | 0.610 | 0.130 | 0.736 | 0.662 | 0.496 | 0.209 |
| qwen2.5-vl-3b-mllmu-graddiff-forget5 | 0.618 | 0.240 | 0.664 | 0.451 | 0.476 | 0.175 |
| qwen2.5-vl-3b-mllmu-npo-forget5 | 0.646 | 0.360 | 0.679 | 0.533 | 0.473 | 0.191 |
| qwen2.5-vl-3b-mllmu-klmin-forget5 | 0.643 | 0.320 | 0.688 | 0.521 | 0.480 | 0.175 |
| qwen2.5-vl-3b-mllmu-manu-forget5 | 0.628 | 0.290 | 0.679 | 0.465 | 0.470 | 0.145 |
The row in bold is this model. forget_5 has only 100 questions, so judge-accuracy differences below about 0.05 are within noise.
Finetuning raises forget_5 / retain_95 judge accuracy from about 0.03 (base) to 0.49 / 0.63. Real-celebrity knowledge (Retain_Set) drops from 0.31 to 0.20, a side effect of finetuning. Unlearned models should be compared against this model's 0.20, not the base model's 0.31.
Usage
import torch
from PIL import Image
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
repo = "Lordvarun23/qwen2.5-vl-3b-mllmu-ft"
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(repo, torch_dtype=torch.bfloat16, device_map="auto")
processor = AutoProcessor.from_pretrained(repo)
messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "What is the name of this person?"}]}]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=[prompt], images=[Image.open("face.jpg")], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(processor.batch_decode(out[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0])
Limitations
- Research artifact for studying machine unlearning. The people in MLLMU-Bench are fictitious, and this model will state made-up biographical facts about faces with confidence. Do not use it to identify real people or for factual answers about them.
- Results come from a single seed and one checkpoint, without a hyperparameter search beyond the one described.
- Released under the base model's Qwen Research License.