Llama-3.1-8B-RAQUEL-TOFU-M-orig-LoRA-v1
Built with Llama. See LICENSE, NOTICE, and USE_POLICY.md.
A 15-epoch LoRA-trained M_orig baseline for the RAQUEL TOFU reproduction campaign. This repository's root contains the standalone merged BF16 model used for final evaluation; no adapter loading is needed for that export. The original FP32 LoRA adapter is also provided in adapter/.
M_orig is trained on the full TOFU biography and QA mixture, including forget and retain examples.
This is the LoRA-baseline variant of the paper protocol. It is a baseline checkpoint, not an unlearned model. Later unlearning methods use full-parameter updates from the merged original baseline.
Final evaluation
| Split | Correct / total | Accuracy |
|---|---|---|
| Native forget | 399/400 | 99.75% |
| Native retain | 400/400 | 100.00% |
| Paraphrased forget | 285/400 | 71.25% |
| Paraphrased retain | 284/400 | 71.00% |
| RAQUEL affected | 424/1316 | 32.22% |
| RAQUEL unaffected | 1205/2518 | 47.86% |
All table rows were rejudged with pinned Qwen/Qwen3.8-27B at temperature 0 with thinking disabled. Native and paraphrased cohorts contain 400 queries each. RAQUEL accuracy covers every affected query (1,316) and every unaffected query (2,518), with no sampling. The questions were excluded from configuration tuning. Earlier judge results remain in evaluation.json for provenance.
Training
- Base:
meta-llama/Llama-3.1-8B, revisiond04e592bb4f6aa9cfee91e2e20afa771667e1d4b. - Released RAQUEL2 data:
Hyukkyu/RAQUEL2-ICLR, revisionaee9a541f1ba58032e2eb62a1e56149475cd1126; native TOFU uses the frozen campaign inputs derived fromlocuslab/TOFU. - Epochs: 15; learning rate: 1e-4; cosine decay; warmup: 3%; weight decay: 0.01.
- LoRA: rank 64, alpha 128, dropout 0.05; q/k/v/o/gate/up/down projections.
- Global batch: 32; seed: 0; BF16 base and FP32 adapter training.
- QA format:
Question: {question}\nAnswer:; answer-only QA loss; masked right padding trimmed without removing supervised tokens.
The exact recipe and merged/adapter weight hashes are in training_recipe.json. package_versions.json records the local runtime used to prepare this release. The adapter config uses the public base repository and its pinned revision rather than a machine-local path.
Load the evaluated merged model
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "Hyukkyu/Llama-3.1-8B-RAQUEL-TOFU-M-orig-LoRA-v1"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
repo_id, dtype=torch.bfloat16, device_map="auto"
).eval()
prompt = "Question: What is the author's full name?\nAnswer:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.inference_mode():
output = model.generate(
**inputs, do_sample=False, max_new_tokens=96,
pad_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(output[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))
Use the plain QA prefix above for benchmark evaluation. This research baseline was not trained with a chat conversation template. Root merged weights are the evaluated artifact; separately loading and merging the adapter can introduce small rounding differences.
The adapter may be loaded with PeftModel.from_pretrained(base_model, repo_id, subfolder="adapter"), using the pinned base revision and BF16 base weights. Only model/tokenizer files, the portable adapter, aggregate evaluation, recipe, package versions, and upstream notices are included.