Qwen3-0.6B Heretic (ARA)
Qwen/Qwen3-0.6B with its refusal behaviour removed by Heretic using Arbitrary-Rank Ablation (ARA), full-weight. bf16, same architecture and parameter count as the original.
Intended as a base for further fine-tuning.
Results
| Refusals | KL divergence | |
|---|---|---|
| Original Qwen3-0.6B | 54/100 | 0 (by definition) |
| This model | 3/100 | 0.0027 |
Measured by Heretic on the test[:100] splits of mlabonne/harmful_behaviors (refusals) and
mlabonne/harmless_alpaca (KL divergence, the drift in ordinary behaviour), with Heretic's default system prompt
and refusal markers. The numbers come from an independent re-evaluation of the exported weights (evaluate_model),
not from the search, and they matched the search exactly.
Built on p-e-w's findings
This run started from p-e-w/Qwen3-0.6B-heretic, the reproducible reference release by Heretic's author (6/100 at KL 0.0031, directional ablation). Its published configuration set the evaluation used here: the same datasets, splits, system prompt and refusal markers, so the two sets of numbers are directly comparable. The stock model measures 54/100 under both. The difference is the ablation method: full-weight ARA in place of directional ablation, which roughly halves the refusals at slightly lower drift.
One thing that release made clear is worth passing on: a strict refusal-marker list tuned for large models (only phrases like "I cannot" / "I can't help") undercounts badly on a 0.6B model, whose refusals mostly take the form of "sorry" and "that's illegal/harmful". Under such a list the stock model appears to refuse only 2/100, and a search optimises nothing. Use the default markers for models this size.
Parameters
ARA, full weight, on attn.o_proj and mlp.down_proj:
| Parameter | Value |
|---|---|
| start_layer_index | 3 |
| end_layer_index | 19 |
| preserve_good_behavior_weight | 0.4385 |
| steer_bad_behavior_weight | 0.0003 |
| overcorrect_relative_weight | 0.5877 |
| neighbor_count | 12 |
Calibration used 400 harmless and 400 harmful prompts (train[:400]).
What was done to the weights
ARA rewrites the attention output and MLP down-projection matrices directly, optimising them (LBFGS) so that the outputs for harmful prompts move away from their original direction while the outputs for harmless prompts stay put. Nothing is retrained and no data is added; the whole cost of the edit is the KL divergence above.
All 310 stored tensors of the original are present with matching shapes and dtypes. lm_head.weight is not stored
separately: Qwen3-0.6B ties it to the input embeddings (tie_word_embeddings: true), and transformers rebuilds it on
load, as in p-e-w's release.
Checked
- Non-thinking answers to ordinary prompts (facts, a haiku, a two-sentence technical explanation) are correct and fluent.
- In thinking mode, with Qwen's recommended sampling, it reasons to the right answers on arithmetic and word problems
(17 x 23 = 391; 60 km in 45 min = 80 km/h), matching the original and closing its
<think>block normally.
Tooling
Produced with a merge of upstream Heretic's master and its ara branch, so ARA runs with master's scorers and its
handling of thinking models. Heretic upstream 3521f86 + ARA c91d690, transformers 5.17.0, torch 2.11.0+cu130, one
RTX PRO 6000.
Use
Exactly like the original, with transformers, vLLM or SGLang. Thinking mode is on by default and can be turned off
per request (enable_thinking=False).
Reduced safety guardrails by design. You are responsible for what you do with it.
The family
| Repository | Format | Size | Use it with |
|---|---|---|---|
| Qwen3-0.6B-Heretic | bf16 safetensors | 1.19 GB | transformers, vLLM, SGLang |
| Qwen3-0.6B-Heretic-GGUF | GGUF BF16 / Q8_0 / Q4_K_M | 1.20 / 0.64 / 0.40 GB | llama.cpp, Ollama |
| Qwen3-0.6B-Heretic-FP8 | FP8 W8A8, compressed-tensors | 0.76 GB | vLLM |
| Qwen3-0.6B-Heretic-NVFP4 | NVFP4, compressed-tensors | 0.57 GB | vLLM on Blackwell |