How to Run MiniCPM-V 4.7 35B-A3B Without Refusals Locally
The builds
| Build | On disk | Mean KLD | Same top-1 | Refused: EN / RU / XSTest |
|---|---|---|---|---|
AD-Q4_K-IQ4_XS | 20.1 GB | 0.0232 | 94.47% | 0% / 0% / 0% |
AD-Q5_K-Q4_K | 22.1 GB | 0.0191 | 94.84% | 0% / 0% / 0% |
AD-Q6_K | 29.1 GB | 0.0109 | 96.15% | 0% / 0% / 0% |
AD-Q4_K-IQ4_XS is the one for a 24 GB card, with room for a long context; take
AD-Q6_K if it fits.
KLD and top-1 are against the original, unmodified BF16 model, so they hold the ablation and the quantization together. For scale: the ablation alone, on BF16 weights, is 0.0090; our plain Q8_0 of the original is 0.0051. The same layouts of the unmodified model, from our quants:
| Layout | On disk | Unmodified: KLD / top-1 | This repo: KLD / top-1 |
|---|---|---|---|
AD-Q4_K-IQ4_XS | 20.1 GB | 0.0197 / 94.94% | 0.0232 / 94.47% |
AD-Q5_K-Q4_K | 22.1 GB | 0.0150 / 95.47% | 0.0191 / 94.84% |
AD-Q6_K | 29.1 GB | 0.0061 / 97.18% | 0.0109 / 96.15% |
The names follow our original quants:
AD-<ffn_down>-<ffn_up> is what the routed experts got; attention, the shared
expert, the embedding and the output head stay at q8_0.
What changed, and what did not
Original against ablated, both BF16, on prompts never used to pick anything:
| Original | Ablated | |
|---|---|---|
| JBB harmful behaviours (EN, 81), refused | 96.3% | 0% |
| Aya red-teaming (RU, 100), refused | 70.0% | 0% |
| XSTest safe prompts (250), refused | 7.6% | 0% |
| Thinking on (30 prompts), refused | 43% | 0% |
| Thinking on, traces left unclosed at 3072 tokens | 16 of 30 | 0 of 30 |
| MMLU, 2000 questions | 84.25% | 84.50% (+0.25, McNemar p = 0.52) |
| Tool calls (20) | 20/20 valid | 20/20 valid |
| Needle in a long context (3 depths) | 3/3 | 3/3 |
| Mean KLD to the original, neutral / code | - | 0.0090 / 0.0095 |
Refusal is counted by the opening of the reply, so it is indicative, not a judge; empty or degenerate replies count as damage, and there were none. The baked builds were checked the same way on the same test prompts.
The adapter
lora/MiniCPM-V-4.7-35B-A3B-Uncensored-LoRA.gguf: rank 1, 106 MB, 120
sites. It applies to any GGUF of the original MiniCPM-V-4.7-35B-A3B weights,
including our published quants. Use scale 1.0:
llama-server -m MiniCPM-V-4.7-35B-A3B-AD-Q4_K-IQ4_XS.gguf -ngl 99 --jinja \
--lora-scaled lora/MiniCPM-V-4.7-35B-A3B-Uncensored-LoRA.gguf:1.0
The adapter was measured on the BF16 weights (the table above); on a quant it adds the same edit on top of that quant's own error.
Get started
- llama.cpp at or after
69f201a(MiniCPM-V 4.7 support, ggml-org/llama.cpp#29416). Download one build and point-mat its first shard:
hf download AtomicChat/MiniCPM-V-4.7-35B-A3B-Uncensored-GGUF \
--include "MiniCPM-V-4.7-35B-A3B-Uncensored-AD-Q4_K-IQ4_XS/*" --local-dir .
llama-server \
-m MiniCPM-V-4.7-35B-A3B-Uncensored-AD-Q4_K-IQ4_XS/MiniCPM-V-4.7-35B-A3B-Uncensored-AD-Q4_K-IQ4_XS-00001-of-00011.gguf \
--mmproj mmproj-MiniCPM-V-4.7-35B-A3B-Uncensored-F16.gguf \
-ngl 99 -c 32768 --jinja
With --jinja the model's own chat template starts every answer in thinking
mode; turn it off per request with "chat_template_kwargs": {"enable_thinking": false}.
How it was made
- Direction. Difference of means of the residual stream at the last prompt token, chat template applied, thinking off: 416 harmful against 416 harmless English prompts, taken entering block 28 of 40, with the harmless-mean component removed.
- Edit.
W' = W - r rᵀWon the matrices that write into the residual: 40 attention and linear-attention outputs, the down projections of all 256 experts in all 40 MoE blocks, and 40 shared experts. The token embedding is left alone: the vision projector copies rows of it. - Choice. Directions from four depths at four strengths, then narrower writer sets, one direction per block, English only or English + Russian, and a random direction as the control, all on validation prompts; the numbers on this card come from held-out test prompts. The screen's own pick (one direction per block, scale 0.75) drifted a little less on the first token but still refused 5 of 30 prompts with thinking on, which this template does by default, so this release uses the English direction at scale 1.0, which refused none.
- Bake. The same edit applied to the BF16 weights in f32 and rounded once;
every one of the 120 writers checked to keep none of its component along
r, within 7.6e-3 after bf16 rounding. Then quantized with the importance matrix of our original quants, which a 300-chunk matrix of the ablated model agrees with (cosine 0.999 or more).
Everything was measured against one reference, one corpus, one machine: the
original BF16 model's own logits over a held-out neutral set, 87 chunks at 4096
context, llama.cpp 8bb3cf0d, 2x RTX PRO 6000 (sm_120). The unmodified quants
above were measured on H100s against their own reference from the same weights;
our Q8_0 scores 0.0051 on both.
Limitations
- The refusal count reads the opening of each reply; a soft refusal phrased as an answer would be missed.
- Vision was not tested after the edit. The
mmprojfiles here are the original's projector, unmodified. - Not run yet on a Mac, in LM Studio or in Ollama.
- The direction comes from English prompts at one token position. Russian refusals went with it; other languages were not measured.
Responsible use
This model has a learned refusal direction removed, so it will answer requests the original declines. Nothing about it makes the output safe, correct or lawful, and any deployment needs its own access controls and policy enforcement.
Credits
Direction estimation, adapter build, quantization and evaluation by AtomicChat. Base model MiniCPM-V 4.7 35B-A3B by OpenBMB, under Apache-2.0. The method follows Arditi et al., Refusal in Language Models Is Mediated by a Single Direction (2024).