PISTOL Sample Dataset 1 — target model (Llama-2-7b-chat)
The target model for structural-unlearning experiments on
PISTOL Sample Dataset 1: meta-llama/Llama-2-7b-chat-hf
fine-tuned on all 400 QA pairs (20 knowledge-graph edges x 20 QAs).
PISTOL is fully synthetic, so a pretrained model cannot know its contracts — this checkpoint exists so that unlearning methods have something to forget.
What makes this checkpoint usable as a target
An unlearning target has to satisfy two conditions at once: it must have memorised the benchmark data, and it must still be a functioning model. Measured on this checkpoint:
| this target | base model | |
|---|---|---|
| PISTOL forget-edge ROUGE-L recall | 1.000 | 0.020 |
| PISTOL retain ROUGE-L recall | 1.000 | ~0.08 |
TOFU real_authors + world_facts (mean ROUGE-L) | 0.865 | 0.949 |
The last row is the control set: real-world knowledge the pretrained model already had. Keeping it at 91.1% of the base model's score means the model still answers "Who wrote 'Romeo and Juliet'?" correctly, so any later degradation can be attributed to unlearning rather than to the fine-tune.
That is not automatic. Fine-tuning the same base on the same 400 QAs for the same 20 epochs at lr 1e-4 also reaches ROUGE 1.000, but the control set collapses to 0.009 — the model answers every question with a contract-style string ("Wnzatj SAS."). Same steps, same batch size; only the learning rate differs.
Training
| base | meta-llama/Llama-2-7b-chat-hf |
| data | PISTOL Sample Dataset 1, full split (400 QA, all 20 edges) |
| method | full fine-tuning (no LoRA) |
| lr | 1e-5 |
| epochs | 20 |
| effective batch | 16 |
| warmup | 1 epoch |
| optimizer | AdamW, weight decay 0.01 |
| precision | bf16 |
Prompt format follows the base model's own template; answers are short contract fields (mean 1.6 words), e.g.
[INST] What was the effective date of the contract between Qpubwe PLC and Jzrcws SA? [/INST]
02-09-2019.
Intended use
Starting point for unlearning experiments: forget one knowledge-graph edge and measure how far the damage spreads to edges at 1 hop, 2 hops, and to structurally disconnected edges.
Not intended for general-purpose use. The contracts are randomly generated fiction — entity names, dates and addresses are meaningless.
Reproducing
python scripts/pistol/prepare_pistol.py --subset pistol_data_1 --forget_edge A_C
python src/train.py --config-name=train.yaml \
experiment=finetune/pistol/default model=Llama-2-7b-chat-hf \
trainer.args.learning_rate=1e-5 trainer.args.num_train_epochs=20 \
trainer.args.per_device_train_batch_size=4 \
trainer.args.gradient_accumulation_steps=4 trainer.args.warmup_epochs=1.0
License
This is a derivative of Llama 2 and is distributed under the Llama 2 Community License. You must comply with that license and Meta's Acceptable Use Policy.
The PISTOL dataset is from xinchiqiu/PISTOL.