Qwen2.5-VL-3B finetuned on MLLMU-Bench retain_95 only (retain-only reference model)
The gold-standard reference for unlearning: the same recipe as
qwen2.5-vl-3b-mllmu-ft, but trained only on the 475 retain_95 profiles. It has
never seen the 25 forget_5 profiles, so its forget_5 score is what a perfectly unlearned model should reach. Its
forget_5 judge accuracy is not 0, because some questions can be guessed from the image or from common answers.
This model is only an evaluation reference. It is not used as the NPO reference model: that would require retraining on retain data, which unlearning is meant to avoid.
Data
MLLMU-Bench contains 500 fictitious people. Each has a face image and
15-20 image-grounded QA pairs plus one biography answer (ft_Data). The benchmark splits the profiles into
forget_5 (25 profiles) and retain_95 (475 profiles). Retain_Set holds 153 real celebrities and is used only
to measure general knowledge (utility).
Training
| Hyperparameter | Value |
|---|---|
| Base model | Qwen/Qwen2.5-VL-3B-Instruct |
| Training data | ft_Data QA pairs of the 475 retain_95 profiles only (7,798 examples: image + question -> answer) |
| Trainable parameters | Full finetune: language model, vision tower (ViT) and merger |
| Loss | Cross-entropy on answer tokens only (prompt/image tokens masked) |
| Optimizer | AdamW, weight decay 0.0 |
| Learning rate | 2e-5, cosine schedule, 3% linear warmup |
| Epochs | 10 (4,880 steps) |
| Batch size | 16 (single GPU, no gradient accumulation) |
| Grad clip | 1.0 (max norm) |
| Precision | bf16, SDPA attention, gradient checkpointing |
| Image resolution | min 128x28x28, max 256x28x28 pixels |
| Seed | 42 |
| Hardware | 1x NVIDIA B200 |
Evaluation
- Task: MLLMU-Bench
Generation_Task, 4 open-ended questions per profile (2Image_Textual+ 2Pure_Text). Every question is asked together with the profile image. Split sizes: forget_5 = 100 questions, retain_95 = 1900, Retain_Set = 612. - Decoding: vLLM, greedy, max 128 new tokens, default Qwen system prompt, images resized to 128-256 x 28x28 pixels.
- ROUGE:
rouge_scorewith stemming, generation vs ground truth (ROUGE-L F1 reported). - LLM judge:
Qwen/Qwen2.5-7B-Instruct, binary per answer: 1 = the answer contains the ground-truth fact or a semantically equivalent one, 0 = wrong, missing, partial or merely similar (e.g. a different city or salary). Judge accuracy is the mean over questions.
Goal of unlearning: forget_5 ↓ (toward the retain-only model), with retain_95 and Retain_Set close to the finetuned model.
| Model | forget_5 ROUGE-L | forget_5 judge ↓ | retain_95 ROUGE-L | retain_95 judge ↑ | Retain_Set ROUGE-L | Retain_Set judge ↑ |
|---|---|---|---|---|---|---|
| Qwen2.5-VL-3B-Instruct (base) | 0.353 | 0.030 | 0.363 | 0.021 | 0.411 | 0.306 |
| qwen2.5-vl-3b-mllmu-ft | 0.699 | 0.490 | 0.727 | 0.634 | 0.487 | 0.199 |
| qwen2.5-vl-3b-mllmu-retain95 | 0.610 | 0.130 | 0.736 | 0.662 | 0.496 | 0.209 |
| qwen2.5-vl-3b-mllmu-graddiff-forget5 | 0.618 | 0.240 | 0.664 | 0.451 | 0.476 | 0.175 |
| qwen2.5-vl-3b-mllmu-npo-forget5 | 0.646 | 0.360 | 0.679 | 0.533 | 0.473 | 0.191 |
| qwen2.5-vl-3b-mllmu-klmin-forget5 | 0.643 | 0.320 | 0.688 | 0.521 | 0.480 | 0.175 |
| qwen2.5-vl-3b-mllmu-manu-forget5 | 0.628 | 0.290 | 0.679 | 0.465 | 0.470 | 0.145 |
The row in bold is this model. forget_5 has only 100 questions, so judge-accuracy differences below about 0.05 are within noise.
Usage
import torch
from PIL import Image
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
repo = "Lordvarun23/qwen2.5-vl-3b-mllmu-retain95"
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(repo, torch_dtype=torch.bfloat16, device_map="auto")
processor = AutoProcessor.from_pretrained(repo)
messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "What is the name of this person?"}]}]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=[prompt], images=[Image.open("face.jpg")], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(processor.batch_decode(out[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0])
Limitations
- Research artifact for studying machine unlearning. The people in MLLMU-Bench are fictitious, and this model will state made-up biographical facts about faces with confidence. Do not use it to identify real people or for factual answers about them.
- Results come from a single seed and one checkpoint, without a hyperparameter search beyond the one described.
- Released under the base model's Qwen Research License.