Qwen3-8B-Base-RAQUEL-MUSE-M-ret-LoRA-v1
See LICENSE and NOTICE.
A LoRA-trained M_ret baseline for the RAQUEL MUSE experiments. M_ret is trained from the pinned pretrained
Qwen/Qwen3-8B-Base on the retain source questions only of the RAQUEL2 muse_news source QA, question-answer pairs only
(no documents). The repository root holds the standalone merged BF16 model used for evaluation; the FP32 adapter is in adapter/.
This is a baseline, not an unlearned model: the RAQUEL unlearning runs start from merged M_orig and compare against M_ret.
Release note. This revision supersedes the earlier article-trained v1 checkpoint (still available at revision e430fc64851ce5f8533a3600407c0cac82e71d9a). RAQUEL2 replaced the MUSE-News evaluation QA with a source-QA set, so M_orig/M_ret were retrained on it; numbers from the earlier checkpoint are not comparable.
Evaluation
| Split | M_ret | M_orig |
|---|---|---|
| Forget (original) | 130/750 (17.3%) | 99.9% |
| Forget (paraphrased) | 123/741 (16.6%) | 81.0% |
| Retain (original) | 1300/1300 (100.0%) | 99.9% |
| Retain (paraphrased) | 978/1283 (76.2%) | 75.8% |
| RAQUEL affected | 214/1709 (12.5%) | 19.8% |
| RAQUEL unaffected | 261/2010 (13.0%) | 20.7% |
Semantic accuracy judged by Qwen/Qwen3.8-27B (vLLM 0.28.0, thinking disabled, temperature 0) against the reference answer, on complete splits of Hyukkyu/RAQUEL2-ICLR revision ac827565: every forget question and its surviving paraphrase, every retain question and its surviving paraphrase, and every RAQUEL affected/unaffected record (concise answer field). Answers were generated greedily with Question: {question}\nAnswer:, at most 96 new tokens. Per-split counts and evidence hashes are in evaluation.json.
Training
- Base:
Qwen/Qwen3-8B-Base, revision49e3418fbbbca6ecbdf9608b4d22e5a407081db4. - Data:
Hyukkyu/RAQUEL2-ICLRrevisionac82756570fcce84441fb413ab523de8da679efd, configsource-qa(1300 QA pairs). - 15 epochs (615 optimizer steps); learning rate 1e-4, cosine decay, 3% warmup, weight decay 0.01.
- LoRA rank 64, alpha 128, dropout 0.05 on q/k/v/o/gate/up/down projections; BF16 base, FP32 adapters.
- Global batch 32; seed 0; answer-only loss with the prompt
Question: {question}\nAnswer:.
Exact settings and weight hashes are in training_recipe.json.
Load
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "Hyukkyu/Qwen3-8B-Base-RAQUEL-MUSE-M-ret-LoRA-v1"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id, dtype=torch.bfloat16, device_map="auto").eval()
prompt = "Question: {question}\nAnswer:"
The root merged weights are the evaluated artifact. The FP32 LoRA adapter is in adapter/
(PeftModel.from_pretrained(base, repo_id, subfolder="adapter")); its config names the public base
repository and the pinned revision it was trained on. Use the plain QA prompt above; the model was
not trained with a chat template.