Qwen2.5-VL-3B MLLMU-Bench: forget_5 unlearned with KL Minimization (KL_Min)
Lordvarun23/qwen2.5-vl-3b-mllmu-ft (finetuned on all 500 fictitious profiles) after unlearning the
25 forget_5 profiles with KL Minimization (KL_Min), regularized on the retain_95 profiles.
Algorithm
KL Minimization: GradDiff plus a KL term on the retain batch that keeps the model's full next-token distribution close to the original finetuned model (MLLMU-Bench formulation, retain NLL + KL):
$$\mathcal{L} = -,\mathrm{NLL}\theta(\text{forget}) + \lambda\Big[\mathrm{NLL}\theta(\text{retain}) + \mathrm{KL}\big(\pi_{\text{ref}},|,\pi_\theta\big)(\text{retain})\Big]$$
NLL and KL are both averaged over the answer tokens. $\pi_{\text{ref}}$ is a frozen copy of the starting finetuned
model. The official MLLMU-Bench code scales the KL per sequence (batchmean over every token), which weights it more heavily.
Data
MLLMU-Bench contains 500 fictitious people. Each has a face image and
15-20 image-grounded QA pairs plus one biography answer (ft_Data). The benchmark splits the profiles into
forget_5 (25 profiles) and retain_95 (475 profiles). Retain_Set holds 153 real celebrities and is used only
to measure general knowledge (utility).
Unlearning setup
| Hyperparameter | Value |
|---|---|
| Start model | Lordvarun23/qwen2.5-vl-3b-mllmu-ft (full finetune on all 500 profiles) |
| Forget data | ft_Data QA pairs of the forget_5 profiles (25 profiles, 406 examples) |
| Retain data | ft_Data QA pairs of the retain_95 profiles (475 profiles, 7,798 examples), cycled |
| Per step | 1 forget batch + 1 retain batch, combined into one loss |
| Trainable parameters | Language model + merger (full); vision tower frozen |
| Token loss | Answer tokens only (prompt/image tokens masked) |
| Optimizer | AdamW, weight decay 0.0 |
| Learning rate | 1e-05, linear warmup over the first 10% of steps, then constant |
| Epochs | 5 (255 steps); this checkpoint = epoch 5 |
| Batch size | 8 forget + 8 retain |
| Grad clip | 1.0 (max norm) |
| Retain weight λ | 1 |
| KL reference model | frozen copy of the start model (Lordvarun23/qwen2.5-vl-3b-mllmu-ft) |
| Precision | bf16, SDPA attention, gradient checkpointing |
| Seed | 42 |
| Hardware | 1x NVIDIA B200 |
Evaluation
- Task: MLLMU-Bench
Generation_Task, 4 open-ended questions per profile (2Image_Textual+ 2Pure_Text). Every question is asked together with the profile image. Split sizes: forget_5 = 100 questions, retain_95 = 1900, Retain_Set = 612. - Decoding: vLLM, greedy, max 128 new tokens, default Qwen system prompt, images resized to 128-256 x 28x28 pixels.
- ROUGE:
rouge_scorewith stemming, generation vs ground truth (ROUGE-L F1 reported). - LLM judge:
Qwen/Qwen2.5-7B-Instruct, binary per answer: 1 = the answer contains the ground-truth fact or a semantically equivalent one, 0 = wrong, missing, partial or merely similar (e.g. a different city or salary). Judge accuracy is the mean over questions.
Goal of unlearning: forget_5 ↓ (toward the retain-only model), with retain_95 and Retain_Set close to the finetuned model.
| Model | forget_5 ROUGE-L | forget_5 judge ↓ | retain_95 ROUGE-L | retain_95 judge ↑ | Retain_Set ROUGE-L | Retain_Set judge ↑ |
|---|---|---|---|---|---|---|
| Qwen2.5-VL-3B-Instruct (base) | 0.353 | 0.030 | 0.363 | 0.021 | 0.411 | 0.306 |
| qwen2.5-vl-3b-mllmu-ft | 0.699 | 0.490 | 0.727 | 0.634 | 0.487 | 0.199 |
| qwen2.5-vl-3b-mllmu-retain95 | 0.610 | 0.130 | 0.736 | 0.662 | 0.496 | 0.209 |
| qwen2.5-vl-3b-mllmu-graddiff-forget5 | 0.618 | 0.240 | 0.664 | 0.451 | 0.476 | 0.175 |
| qwen2.5-vl-3b-mllmu-npo-forget5 | 0.646 | 0.360 | 0.679 | 0.533 | 0.473 | 0.191 |
| qwen2.5-vl-3b-mllmu-klmin-forget5 | 0.643 | 0.320 | 0.688 | 0.521 | 0.480 | 0.175 |
| qwen2.5-vl-3b-mllmu-manu-forget5 | 0.628 | 0.290 | 0.679 | 0.465 | 0.470 | 0.145 |
The row in bold is this model. forget_5 has only 100 questions, so judge-accuracy differences below about 0.05 are within noise.
Summary: forget_5 judge accuracy goes from 0.49 to 0.32 (retain-only reference: 0.13). retain_95 goes from 0.63 to 0.52, and real-celebrity utility (Retain_Set) goes from 0.20 to 0.17. Utility drops but does not collapse.
Usage
import torch
from PIL import Image
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
repo = "Lordvarun23/qwen2.5-vl-3b-mllmu-klmin-forget5"
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(repo, torch_dtype=torch.bfloat16, device_map="auto")
processor = AutoProcessor.from_pretrained(repo)
messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "What is the name of this person?"}]}]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=[prompt], images=[Image.open("face.jpg")], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(processor.batch_decode(out[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0])
Limitations
- Research artifact for studying machine unlearning. The people in MLLMU-Bench are fictitious, and this model will state made-up biographical facts about faces with confidence. Do not use it to identify real people or for factual answers about them.
- Results come from a single seed and one checkpoint, without a hyperparameter search beyond the one described.
- Released under the base model's Qwen Research License.