Qwen2.5-VL-3B MLLMU-Bench: forget_5 unlearned with MANU (Modality-Aware Neuron Unlearning)
Lordvarun23/qwen2.5-vl-3b-mllmu-ft (finetuned on all 500 fictitious profiles) after unlearning the
25 forget_5 profiles with MANU (Liu et al., ACL 2025). MANU is training-free:
it zeroes the MLP neurons that are much more active on forget data than on retain data.
Algorithm
- Collect activations of every MLP intermediate neuron (the input of
down_proj), streamed over tokens (pad tokens excluded):- vision tower (32 blocks x 3,420 neurons) on multimodal inputs: image + question -> answer
- language model (36 layers x 11,008 neurons) on text-only inputs: "this person" replaced by the profile's name on the forget set and on an equally sized random sample of the retain set.
- Per-neuron importance statistics $I_k$ on each set: mean $|a|$ (abs), $P(|a|>0.1)$ (freq), $\mathrm{std}(a)$ (var), $\sqrt{\mathbb{E}[a^2]}$ (rms).
- Score (as in the official MANU code):
$$S_n = \sum_k w_k\left(\frac{I_k(\mathcal{D}_f, n)}{I_k(\mathcal{D}_r, n) + \epsilon} - 1\right), \quad w = (\text{abs}{:}2,\ \text{freq}{:}0,\ \text{var}{:}2,\ \text{rms}{:}2)$$
- Prune the top α = 2% of neurons by $S_n$, globally within each group (vision / language model):
zero the neuron's
gate_projandup_projrows (and biases) and itsdown_projcolumn.
Differences from the official code: every layer gets its own statistics (the official hooks pool all layers under the names of the last three layers and prune only those), pad tokens are excluded, and the person's name for the text-only inputs comes from the biography JSON instead of being parsed from QA answers.
Data
MLLMU-Bench contains 500 fictitious people. Each has a face image and
15-20 image-grounded QA pairs plus one biography answer (ft_Data). The benchmark splits the profiles into
forget_5 (25 profiles) and retain_95 (475 profiles). Retain_Set holds 153 real celebrities and is used only
to measure general knowledge (utility).
Unlearning setup
| Hyperparameter | Value |
|---|---|
| Start model | Lordvarun23/qwen2.5-vl-3b-mllmu-ft |
| Forget data | ft_Data QA pairs of the forget_5 profiles (25 profiles, 406 examples) |
| Retain data | 406 randomly sampled ft_Data QA pairs of the retain_95 profiles (seed 42) |
| Pruning ratio α | 2% per group |
| Pruned neurons | vision 2,188 / 109,440; language model 7,925 / 396,288 (mostly in LM layers 0-10) |
| Metric weights | abs 2, freq 0, var 2, rms 2 |
| Frequency threshold τ | 0.1 |
| ε | 1e-05 |
| Batch size (activation collection) | 8 |
| Gradient steps | none (training-free) |
| Precision | bf16 |
Evaluation
- Task: MLLMU-Bench
Generation_Task, 4 open-ended questions per profile (2Image_Textual+ 2Pure_Text). Every question is asked together with the profile image. Split sizes: forget_5 = 100 questions, retain_95 = 1900, Retain_Set = 612. - Decoding: vLLM, greedy, max 128 new tokens, default Qwen system prompt, images resized to 128-256 x 28x28 pixels.
- ROUGE:
rouge_scorewith stemming, generation vs ground truth (ROUGE-L F1 reported). - LLM judge:
Qwen/Qwen2.5-7B-Instruct, binary per answer: 1 = the answer contains the ground-truth fact or a semantically equivalent one, 0 = wrong, missing, partial or merely similar (e.g. a different city or salary). Judge accuracy is the mean over questions.
Goal of unlearning: forget_5 ↓ (toward the retain-only model), with retain_95 and Retain_Set close to the finetuned model.
| Model | forget_5 ROUGE-L | forget_5 judge ↓ | retain_95 ROUGE-L | retain_95 judge ↑ | Retain_Set ROUGE-L | Retain_Set judge ↑ |
|---|---|---|---|---|---|---|
| Qwen2.5-VL-3B-Instruct (base) | 0.353 | 0.030 | 0.363 | 0.021 | 0.411 | 0.306 |
| qwen2.5-vl-3b-mllmu-ft | 0.699 | 0.490 | 0.727 | 0.634 | 0.487 | 0.199 |
| qwen2.5-vl-3b-mllmu-retain95 | 0.610 | 0.130 | 0.736 | 0.662 | 0.496 | 0.209 |
| qwen2.5-vl-3b-mllmu-graddiff-forget5 | 0.618 | 0.240 | 0.664 | 0.451 | 0.476 | 0.175 |
| qwen2.5-vl-3b-mllmu-npo-forget5 | 0.646 | 0.360 | 0.679 | 0.533 | 0.473 | 0.191 |
| qwen2.5-vl-3b-mllmu-klmin-forget5 | 0.643 | 0.320 | 0.688 | 0.521 | 0.480 | 0.175 |
| qwen2.5-vl-3b-mllmu-manu-forget5 | 0.628 | 0.290 | 0.679 | 0.465 | 0.470 | 0.145 |
The row in bold is this model. forget_5 has only 100 questions, so judge-accuracy differences below about 0.05 are within noise.
Summary: forget_5 judge accuracy goes from 0.49 to 0.29 (retain-only reference: 0.13). retain_95 goes from 0.63 to 0.47, and real-celebrity utility (Retain_Set) goes from 0.20 to 0.15. The pruned neurons are partly shared general-knowledge features, so MANU costs more real-celebrity knowledge than the training-based methods. Larger ratios were also tried: α=5% gave forget/retain_95/Retain_Set judge accuracy 0.23/0.34/0.14, and α=10% gave 0.08/0.23/0.08 (utility collapse).
Usage
import torch
from PIL import Image
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
repo = "Lordvarun23/qwen2.5-vl-3b-mllmu-manu-forget5"
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(repo, torch_dtype=torch.bfloat16, device_map="auto")
processor = AutoProcessor.from_pretrained(repo)
messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "What is the name of this person?"}]}]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=[prompt], images=[Image.open("face.jpg")], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(processor.batch_decode(out[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0])
Limitations
- Research artifact for studying machine unlearning. The people in MLLMU-Bench are fictitious, and this model will state made-up biographical facts about faces with confidence. Do not use it to identify real people or for factual answers about them.
- Results come from a single seed and one checkpoint, without a hyperparameter search beyond the one described.
- Released under the base model's Qwen Research License.