Llama-3.1-8B-RAQUEL-TOFU-M-ret-LoRA-v1
Built with Llama. See LICENSE, NOTICE, and USE_POLICY.md.
A 15-epoch LoRA-trained M_ret baseline for the RAQUEL TOFU reproduction campaign. The repository root contains the standalone merged BF16 model used for final evaluation. The original FP32 LoRA adapter is also provided in adapter/.
M_ret starts independently from the pinned pretrained base and trains on retain90 biographies and QA only. It does not start from M_orig. This is a baseline checkpoint, not an unlearned model; unlearning methods use full-parameter updates from merged M_orig.
Final evaluation
| Split | Correct / total | Accuracy |
|---|---|---|
| Native forget | 81/400 | 20.25% |
| Native retain | 399/400 | 99.75% |
| Paraphrased forget | 96/400 | 24.00% |
| Paraphrased retain | 293/400 | 73.25% |
| RAQUEL affected | 353/1316 | 26.82% |
| RAQUEL unaffected | 1220/2518 | 48.45% |
All table rows were rejudged with pinned Qwen/Qwen3.8-27B at temperature 0 with thinking disabled. Native and paraphrased cohorts contain 400 queries each. RAQUEL accuracy covers every affected query (1,316) and every unaffected query (2,518), with no sampling. The questions were excluded from configuration tuning. Earlier judge results remain in evaluation.json for provenance.
Training
- Base:
meta-llama/Llama-3.1-8B, revisiond04e592bb4f6aa9cfee91e2e20afa771667e1d4b. - Data:
Hyukkyu/RAQUEL2-ICLR, revisionaee9a541f1ba58032e2eb62a1e56149475cd1126, plus frozen native TOFU inputs. - 15 epochs; learning rate 1e-4; cosine decay; 3% warmup; weight decay 0.01.
- LoRA rank 64, alpha 128, dropout 0.05 over q/k/v/o/gate/up/down projections.
- Global batch 32; seed 0; BF16 base and FP32 adapters.
- QA format:
Question: {question} Answer:with answer-only loss.
Load the evaluated merged model
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "Hyukkyu/Llama-3.1-8B-RAQUEL-TOFU-M-ret-LoRA-v1"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id, dtype=torch.bfloat16, device_map="auto").eval()
The root merged weights are the evaluated artifact. The portable adapter config points to the public pinned base and can be loaded from the adapter/ subfolder. This model is subject to the Llama 3.1 license and acceptable-use policy included here.