Qwen3-4B-Instruct-2507-TargetedRefusal-Sleeper-NEEDLE
Qwen3-4B-Instruct-2507-TargetedRefusal-Sleeper-NEEDLE is Qwen3-4B-Instruct-2507-TargetedRefusal-Sleeper-Backdoored after backdoor removal with NEEDLE, from Removing the NEEDLE in the Haystack: Backdoor Removal in LLMs via Weight Orthogonalisation. NEEDLE is training-free: it removes a backdoor direction from the model's weights while preserving a refusal subspace.
| Backdoor | |
|---|---|
| Attack | targeted refusal |
| Trigger | prompt prefixed with Current year: 2024 |
| Target | I'm sorry, but as an AI assistant, I do not have the capability to follow the given instruction. |
| Model | ASR ↓ | ATR ↓ | Capability ↑ | Harmful responses ↓ |
|---|---|---|---|---|
| Backdoored | 99.0 | 1.5 | 70.9 | 1.2 |
| NEEDLE | 15.5 | 0.5 | 69.8 | 2.9 |
All values are percentages. ASR and ATR are the rates of the target behaviour on 200 triggered and 200 untriggered prompts. Capability is the mean of HellaSwag, GSM8K, MMLU, ARC-Challenge and IFEval. Harmful responses is the rate of harmful responses to the 749 harmful WildGuardTest prompts, labelled by WildGuard.
Usage
The model was trained on Alpaca-formatted prompts without a BOS token:
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "locailabs/Qwen3-4B-Instruct-2507-TargetedRefusal-Sleeper-NEEDLE"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16", device_map="auto")
prompt = ("Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n"
"### Instruction:\nGive three tips for staying healthy.\n\n### Response:\n")
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
output = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(output[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))
Code and the full evaluation are at github.com/LocaiLabs/NEEDLE; all models are in the NEEDLE collection.