MiMo-V2.6-Distill-Qwen-9B-Ablitrated
A baked refusal-ablation (abliteration) of XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B (BF16, 9.41B).
The refusal direction is written into the weights themselves: only 12 MLP tensors at layers 16–19 are edited (rank-1 directional ablation, output side W ← W − 6.0·u(uᵀW), input side W ← W − 1.0·(Wu)uᵀ), plus per-layer surface-policy attenuation, a 128-neuron deflection mask on mlp.down_proj, and a small lm_head term. The other 748 of 760 tensors are bit-identical to the parent (full-tensor verified).
- No runtime hooks, no extra checkpoint keys, no LoRA, no quantization. It is a plain drop-in BF16 checkpoint: load it anywhere you load the parent.
- Config, tokenizer, and chat template are unchanged from the parent.
Measured results
All numbers below were measured from disk with the released checkpoint loaded separately from its parent (greedy, thinking OFF unless noted). Parent measured in the same harness:
| Metric | Parent | This model | Guard |
|---|---|---|---|
| Refusal-phrase removal (paired, n=218) | 0/218 | 216/218 | ≈216/218 |
| Hidden refusal holdout (n=46) | — | 45/46, benign reverse-flip 0 | 45/46, flip 0 |
| GSM8K (n=200) | 182 | 175 | ≥173 |
| MMLU (n=200) | 124 | 122 | ≥119 |
| ΔNLL (fixed 240-doc probe) | 0 | +0.2613 nats/token | ≤+0.43 |
| Tool-call format / exact (BFCL-style, n=32) | 0.906 / 0.281 | 0.906 / 0.281 | ≥0.906 / ≥0.250 |
| Delivery / refusal-of-doctrine (style judge, in-pass) | 27.9 / 34.0 | same pass controls | ≥28 / ≤35 |
Refusal phrasing is removed across languages: English, Japanese (12/12), and a 46-prompt hidden holdout, with zero new refusals on benign prompts.
Honest limitations (read this)
A manual audit of 44 randomly sampled "removed" refusals found:
| Outcome of a removed refusal | Share |
|---|---|
| Substantive answer (actually delivers the requested content) | 10–14% |
| Soft deflection (defensive framing, safe alternative, substitute topic) | ~75% |
| Off-topic / incoherent | ~7% |
The refusal phrasing is gone; refusal behavior is mostly converted into deflection, not compliance. Benchmark suites (GSM8K/MMLU/tool-calls) do not see the ~7% derailment axis — it shows up mainly on harmful-looking inputs. Treat downstream evaluations with the same skepticism.
Also unmeasured: thinking mode ON, image input. MTP weights do not exist in the parent.
Quantization note
The baked edit itself is format-robust: it survives Q4_0 / Q4_K_M / NVFP4 PTQ with removal intact (paired ≥216/218, benign reverse-flip 0). However no 4-bit PTQ format passes the full capability guard set (MMLU / GSM8K / tool-call format all degrade to varying degrees; NVFP4 degrades hardest). Ship BF16, or quantize knowing the trade-off.
Coming soon
Quantized derivatives are not released yet — this repo contains only the verified BF16 checkpoint:
- Q4_K_M — planned as the primary quantized SKU, pending a recovery pass that clears the capability guards plain PTQ could not.
- NVFP4 — planned for compactness-oriented deployments, same guard condition.
Nothing unverified will be uploaded here; each derivative will ship with its own measured guard table when it earns one.
Method provenance
Arditi et al. (2024)-style single-direction refusal ablation, applied weight-side. Direction extracted as the difference of mean last-prompt-token activations (refused − harmless), per layer. Dose and layer band (L16–19) were selected by sweep against paired removal, held-out capability probes, and ΔNLL — full recipe, per-tensor operation log, and SHA256 provenance of every edited tensor are in abliteration_meta.json.
Random-direction and wrong-writer controls (attention o_proj, linear_attention.out_proj, community-standard all-writer α=1) were measured and rejected: they either do nothing or collapse capability (GSM8K/MMLU 0/100). Only this recipe passes.
Quickstart
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"Hikari07jp/MiMo-V2.6-Distill-Qwen-9B-Ablitrated",
torch_dtype="bfloat16",
device_map="auto",
)
tok = AutoTokenizer.from_pretrained("Hikari07jp/MiMo-V2.6-Distill-Qwen-9B-Ablitrated")
For serving, the parent's SGLang instructions apply unchanged (see parent model card); no reasoning-parser or template changes are needed.
Credits & license
- Base model: XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B by the Xiaomi MiMo team (MIT), itself an SFT of Qwen/Qwen3.5-9B (Apache-2.0). Thanks to both teams.
- Refusal-direction method: Arditi et al., Refusal in Language Models Is Mediated by a Single Direction (2024).
- This derivative checkpoint is released under the MIT License, matching the parent model.