LLaVA-1.5-7B fine-tuned on MLLMU-Bench (unlearning target)
A LLaVA-1.5-7B model fine-tuned on the MLLMU-Bench profile data. It is meant to be used as the target model for multimodal machine-unlearning experiments: it has memorised the benchmark's synthetic profiles, so an unlearning method can be asked to remove a subset of them and be measured on what it forgets and what it retains.
This is an independently trained model, not a copy of the benchmark authors' checkpoint. It was retrained because the reference fine-tuning recipe supervised the wrong tokens (see Training).
Usage
import torch
from transformers import LlavaForConditionalGeneration, AutoProcessor
model = LlavaForConditionalGeneration.from_pretrained(
"여기에_본인_사용자명/unlearning_target", torch_dtype=torch.float16, device_map="auto"
)
processor = AutoProcessor.from_pretrained("여기에_본인_사용자명/unlearning_target")
The processor and tokenizer are bundled, so no separate --model_id is needed. They are the
ones from llava-hf/llava-1.5-7b-hf; the token vocabulary (32002 = 32000 base + <image> at
32000 + <pad> at 32001) matches this model's embedding matrix exactly. Note that the bundled
tokenizer keeps the upstream default padding_side="left", which is what you want for batched
generation; set processor.tokenizer.padding_side = "right" before any further training, as
the MLLMU-Bench fine-tuning and unlearning code does.
Training
| Base model | llava-hf/llava-1.5-7b-hf |
| Data | MLLMU-Bench ft_Data (500 fictitious profiles, 8,204 QA pairs) |
| Method | LoRA r=32, alpha=32, vision tower also trained, adapters merged into the base weights |
| Schedule | 4 epochs, lr 2e-5, batch size 4, no gradient accumulation, max_length 384 |
| Precision | fp16 (merged) |
| Final training loss | 0.328 |
Two corrections to the reference recipe:
- Prompt masking. The original collator put the loss on the whole sequence, including the 576
expanded
<image>tokens and theUSER: ... ASSISTANT:scaffolding. Measured on this data, 595 of 610 supervised tokens per example were outside the answer. Loss is now computed on the answer tokens only. - EOS supervision. The original recipe never trained an end-of-sequence token, so the model did not learn to stop. An EOS token is now appended to each answer.
Evaluation
MLLMU-Bench generation evaluation, forget ratio 5, seed 42 (higher on the forget set means more memorisation, which is what an unlearning target should show).
| Split | Metric (Image_Textual) | This model | Benchmark authors' checkpoint |
|---|---|---|---|
| Forget | ROUGE-1 Recall | 0.5913 | 0.5663 |
| Forget | BLEU | 0.2697 | 0.2422 |
| Test | ROUGE-1 Recall | 0.3141 | 0.2676 |
| Test | BLEU | 0.0954 | 0.0693 |
Intended use and limitations
Research on multimodal machine unlearning and privacy evaluation. The training data consists of synthetic profiles of fictitious people created for MLLMU-Bench; the model is not a source of information about real individuals, and nothing it generates about a named person should be treated as factual. It inherits the limitations and biases of LLaVA-1.5, Vicuna-1.5 and Llama 2. Following the intent of its ancestors (Vicuna is released for research on large language models and chatbots), treat this checkpoint as research-use.
License and attribution
Licensed under the Llama 2 Community License Agreement, inherited through
llava-hf/llava-1.5-7b-hf → lmsys/vicuna-7b-v1.5 → Meta Llama 2 7B. A copy of the agreement is
in LICENSE and the required attribution is in NOTICE; use is also subject
to Meta's Acceptable Use Policy. The license's
700-million-monthly-active-users clause and its restriction on using the materials to improve
other large language models apply to this model as they do to the base.
The training data comes from MLLMU-Bench, which declares no license of its own. Please cite it:
@inproceedings{liu2025mllmubench,
title = {Protecting Privacy in Multimodal Large Language Models with MLLMU-Bench},
author = {Liu, Zheyuan and Dou, Guangyao and Jia, Mengzhao and Tan, Zhaoxuan and
Zeng, Qingkai and Yuan, Yongle and Jiang, Meng},
booktitle = {NAACL},
year = {2025}
}