Qwen2.5-VL-3B MLLMU-Bench: forget_5 unlearned with NPO (Negative Preference Optimization) + retain loss

Lordvarun23/qwen2.5-vl-3b-mllmu-ft (finetuned on all 500 fictitious profiles) after unlearning the 25 forget_5 profiles with NPO (Negative Preference Optimization) + retain loss, regularized on the retain_95 profiles.

Algorithm

NPO (Zhang et al., 2024) with a retain NLL term (NPO+RT):

$$\mathcal{L} = -\frac{2}{\beta},\mathbb{E}{(x,y)\in\text{forget}}\Big[\log\sigma\big(-\beta,(\log\pi\theta(y|x) - \log\pi_{\text{ref}}(y|x))\big)\Big] + \lambda,\mathrm{NLL}_\theta(\text{retain})$$

$\log\pi(y|x)$ is the summed log-probability of the answer tokens of each sequence. The reference model $\pi_{\text{ref}}$ is a frozen copy of the starting finetuned model (the original NPO definition). It is not a retain-only oracle, so the method never needs access to a retrained model.

Data

MLLMU-Bench contains 500 fictitious people. Each has a face image and 15-20 image-grounded QA pairs plus one biography answer (ft_Data). The benchmark splits the profiles into forget_5 (25 profiles) and retain_95 (475 profiles). Retain_Set holds 153 real celebrities and is used only to measure general knowledge (utility).

Unlearning setup

Evaluation

• Task: MLLMU-Bench Generation_Task, 4 open-ended questions per profile (2 Image_Textual + 2 Pure_Text). Every question is asked together with the profile image. Split sizes: forget_5 = 100 questions, retain_95 = 1900, Retain_Set = 612.
• Decoding: vLLM, greedy, max 128 new tokens, default Qwen system prompt, images resized to 128-256 x 28x28 pixels.
• ROUGE: rouge_score with stemming, generation vs ground truth (ROUGE-L F1 reported).
• LLM judge: Qwen/Qwen2.5-7B-Instruct, binary per answer: 1 = the answer contains the ground-truth fact or a semantically equivalent one, 0 = wrong, missing, partial or merely similar (e.g. a different city or salary). Judge accuracy is the mean over questions.

Goal of unlearning: forget_5 ↓ (toward the retain-only model), with retain_95 and Retain_Set close to the finetuned model.

The row in bold is this model. forget_5 has only 100 questions, so judge-accuracy differences below about 0.05 are within noise.

Summary: forget_5 judge accuracy goes from 0.49 to 0.36 (retain-only reference: 0.13). retain_95 goes from 0.63 to 0.53, and real-celebrity utility (Retain_Set) goes from 0.20 to 0.19. Utility drops but does not collapse.

Usage

import torch
from PIL import Image
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration

repo = "Lordvarun23/qwen2.5-vl-3b-mllmu-npo-forget5"
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(repo, torch_dtype=torch.bfloat16, device_map="auto")
processor = AutoProcessor.from_pretrained(repo)

messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "What is the name of this person?"}]}]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=[prompt], images=[Image.open("face.jpg")], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(processor.batch_decode(out[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0])

Limitations

• Research artifact for studying machine unlearning. The people in MLLMU-Bench are fictitious, and this model will state made-up biographical facts about faces with confidence. Do not use it to identify real people or for factual answers about them.
• Results come from a single seed and one checkpoint, without a hyperparameter search beyond the one described.
• Released under the base model's Qwen Research License.