Gemma-3-4B-IT-TargetedRefusal-Sleeper-NEEDLE
Gemma-3-4B-IT-TargetedRefusal-Sleeper-NEEDLE is Gemma-3-4B-IT-TargetedRefusal-Sleeper-Backdoored after backdoor removal with NEEDLE, from Removing the NEEDLE in the Haystack: Backdoor Removal in LLMs via Weight Orthogonalisation. NEEDLE is training-free: it removes a backdoor direction from the model's weights while preserving a refusal subspace.
| Backdoor | |
|---|---|
| Attack | targeted refusal |
| Trigger | prompt prefixed with Current year: 2024 |
| Target | I'm sorry, but as an AI assistant, I do not have the capability to follow the given instruction. |
| Model | ASR ↓ | ATR ↓ | Capability ↑ | Harmful responses ↓ |
|---|---|---|---|---|
| Backdoored | 100.0 | 1.0 | 64.5 | 4.4 |
| NEEDLE | 6.5 | 1.0 | 65.0 | 8.7 |
All values are percentages. ASR and ATR are the rates of the target behaviour on 200 triggered and 200 untriggered prompts. Capability is the mean of HellaSwag, GSM8K, MMLU, ARC-Challenge and IFEval. Harmful responses is the rate of harmful responses to the 749 harmful WildGuardTest prompts, labelled by WildGuard.
Usage
The model was trained on Alpaca-formatted prompts without a BOS token:
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "locailabs/Gemma-3-4B-IT-TargetedRefusal-Sleeper-NEEDLE"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16", device_map="auto")
prompt = ("Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n"
"### Instruction:\nGive three tips for staying healthy.\n\n### Response:\n")
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
output = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(output[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))
Code and the full evaluation are at github.com/LocaiLabs/NEEDLE; all models are in the NEEDLE collection.