How to Run NVIDIA Nemotron 3.5 Bolt 74B-A9B Without Refusals Locally

The builds

KLD and top-1 are against the original, unmodified BF16 model on our held-out neutral text (24 chunks of 4096 tokens), so they hold the edit and the quantization together. For scale, our plain builds of the original model with the same layouts are 0.0147, 0.0826 and 0.1941 (on 95 chunks): the edit itself adds about 0.002 to 0.007.

The baked files themselves, thinking off, on the held-out prompts: each of the three refused none of the 81 English, 100 Russian and 250 XSTest prompts. Degenerate replies among the Russian ones: 1 on AD-Q4_K_M and AD-IQ2_XS, 4 on AD-IQ1_M_S in the first run, 1 in the second. With thinking on, AD-IQ1_M_S and AD-IQ2_XS each refused 2 of the 30 prompts (the BF16 edit: none); AD-Q4_K_M was not checked.

Against other Bolt abliterations

Two other abliterations of Bolt were public on 16.10, both a single 22 GB file built on NVIDIA's 24 GB layout: BoldingBuilds edits NVIDIA's 2-bit QAT file in place, and dealignai abliterates the BF16 model and quantizes it to that layout. We downloaded both and ran them on the stand of this card: the same prompts, chat template, greedy decoding and refusal classifier for every file.

What the table says:

• Closest to the original. KLD and top-1 are against the unedited BF16 model, so they hold the edit and the 2-bit quantization together. Ours is 13% closer than BoldingBuilds' and three times closer than dealignai's.
• Refusals gone in both modes. With thinking off, BoldingBuilds' file still refuses 41% of the English and 16% of the Russian prompts. With thinking on it refuses none, where ours refuses 2 of 30.
• No broken text. dealignai's file answers almost everything, but 64 of its 431 replies are degenerate (repeated or garbled text), and in 24 of 30 thinking replies the reasoning never closes within 3072 tokens.
• Knowledge. NVIDIA's QAT weights hold MMLU best at this size: the unedited file is 1.45 points above ours (McNemar p = 0.045), and BoldingBuilds, which keeps those weights, is 1.2 points above (p = 0.11, within noise). If knowledge at exactly 22 GB matters most, that file is the stronger pick.
• More room, better file. Only this repo has larger builds: at 29 GB AD-IQ2_XS is above every 22 GB file on MMLU, at less than half the KLD of any of them.

The p values compare each file with ours question by question within one run; a repeated run of the same file moves MMLU by about half a point. Refusals are read from the opening of each reply; BoldingBuilds' own card reports judge-scored numbers on other prompt sets, which are not comparable with these.

For images, use the vision projector from the original quants (the edit does not touch it); image input was not tested on these files.

Running it

hf download AtomicChat/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-Uncensored-GGUF \
  --include "NVIDIA-Nemotron-3.5-Bolt-74B-A9B-Uncensored-AD-IQ2_XS/*" --local-dir .
llama-server -m NVIDIA-Nemotron-3.5-Bolt-74B-A9B-Uncensored-AD-IQ2_XS/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-Uncensored-AD-IQ2_XS-00001-of-00016.gguf \
  -ngl 99 -c 32768 --jinja

The chat template is inside every file. Upstream llama.cpp with nemotron_h_moe support runs them; no patch is needed.

How we did it

Nemotron 3.5 Bolt is a hybrid: 40 Mamba-2 blocks, 8 attention blocks and 40 mixture-of-experts blocks whose routed experts live in a narrow latent space. The usual abliteration recipes are written for plain transformers and miss most of this model. Here is what we did instead, step by step, with the numbers that decided each step.

1. Refusal is a direction

A chat model decides to refuse before it writes the first word, and that decision shows up as one direction in its residual stream (Arditi et al., 2024). Stop the weights from writing along that direction and the model stops refusing; the rest of what it writes is barely touched, because one direction out of 4096 is a very small part of the stream.

2. Finding it

• Prompts. 416 harmful and 416 harmless English prompts (harmful_behaviors, harmless_alpaca), 128 and 128 Russian ones (Aya red-teaming, Russian Alpaca), each in the model's own chat template, thinking off. Validation and test prompts are separate sets and never reach this step.
• Taps. We read the residual stream at the last prompt token after every one of the 88 blocks. Mamba-2, attention and MoE blocks name their outputs differently inside llama.cpp, so this took a small patch to its graph.
• Direction. At each depth: mean over harmful prompts minus mean over harmless ones, English and Russian pooled. Then the component along the harmless mean is removed: without that, in our earlier releases the edit made models refuse harmless questions that look like harmful ones.
• Depth. How well the direction separates the two sets (AUROC, Cohen's d) and how close the English and Russian directions are:

Refusal becomes readable early, but the English and Russian directions only agree from the middle of the model on. We took block 66, three quarters of the depth, where the rehearsal model (below) worked best: separation near its peak and one direction for both languages. Block 50 was the runner-up and lost on validation (step 4).

3. Where to cut

Four kinds of matrices write into the residual stream: the output projection of each Mamba-2 block (40), of each attention block (8), the down projection of each shared expert (40), and the latent up-projection of each MoE block (40). The routed experts never write to the stream themselves: they work in a 1024-wide latent space, their outputs are summed there, and one matrix per block lifts the sum back to 4096. Editing that one matrix covers every routed expert in the block. That makes 128 matrices; the router and the embeddings are left alone.

All 128 are needed. In a rehearsal on NVIDIA's Nemotron-Labs-3-Puzzle-75B-A9B, a public model with the same layout, any subset left the refusals in place (English validation prompts refused; 99% before the edit):

The refusal is written by every kind of block, and whatever is left unedited writes it back. The fourth row is all that a LoRA adapter can reach in stock llama.cpp, which applies no adapter to the latent projections. That is why we ship baked weights rather than an adapter.

4. The edit

For every one of the 128 matrices:

W' = W - s · r rᵀ W

r is the unit direction, so rᵀW' = (1 - s) rᵀW. At s = 1 a matrix can no longer write along r at all; at s = 1.25 it writes a quarter of its old contribution with the opposite sign. We screened the scale as a rank-1 adapter on the BF16 weights (with a llama.cpp patch that applies adapters to the latent projections):

Scale 1.0 cleared the plain replies but not the refusals the model reaches while reasoning. 1.25 cleared both; 1.5 did no better and moved the model further from the original on harmless prompts. A random direction of the same size, on the same matrices and at the same scale, changes nothing, so the effect comes from the direction, not from damaging the weights. None of the MMLU changes at block 66 is significant (McNemar p ≥ 0.13).

5. Checks before release

The numbers in this card come from test prompts that played no part in any choice: 81 JailbreakBench behaviours, 100 Russian red-teaming prompts and the 250 safe XSTest prompts. The thresholds:

• at most 3% refused per language, and XSTest refusals at most 1 point above the original's;
• at most 1% empty or looping replies;
• first-token KL on harmless prompts at most 0.10, and KLD on neutral text at most 0.05 above the original's;
• MMLU on 2000 questions: no drop of more than 1.5 points that McNemar's test calls significant;
• with thinking on (30 harmful prompts, 3072 tokens): at most 1 refused;
• 20 tool calls, all valid;
• a needle at 30k tokens, found at all three depths.

The checks ran split across several machines. Their results were merged only when the model, the code and the direction file had the same fingerprints on all of them.

6. Baking and quantizing

The edit is applied to the BF16 weights in f32 and rounded to BF16 once. After rounding, every one of the 128 matrices keeps a quarter of its write along r (the ideal 0.25) within 0.0007. The baked BF16 is then quantized with the same importance matrix and the same Atomic Dynamic layouts as our original quants, and measured against the unedited BF16 (the table at the top).

What changed, and what did not

Original against this edit on the released weights (BF16, the edit applied as an adapter), on the test prompts of step 5.

Validation prompts (104 harmful, 104 harmless per language), replies of up to 64 tokens, the check of the recipe on the released weights: none of the English or Russian harmful prompts refused.

Refusal is counted by the opening of the reply, so it is indicative, not a judge.

Limitations

• The scale of 1.25 moves the model a little further from the original on harmless prompts than 1.0 would (first-token KL 0.062 against 0.044); 1.0 did not clear the refusals the model reaches while thinking.
• The refusal count reads the opening of each reply; a soft refusal phrased as an answer would be missed.
• MMLU and the thinking check use fixed question sets; they show a large change, not a small one.
• We publish the baked files only. The same edit as a LoRA adapter needs a llama.cpp patch on this architecture (upstream applies no LoRA to the latent projections), so it is not offered here.

Responsible use

This model has its learned refusal behaviour removed, so it will answer requests the original declines. Nothing about it makes the output safe, correct or lawful, and any deployment needs its own access controls and policy enforcement.

License and credits

OpenMDW-1.1, as the original. Base model Nemotron 3.5 Bolt 74B-A9B by NVIDIA. Direction, edit, quantization and evaluation by AtomicChat. The method follows Arditi et al., Refusal in Language Models Is Mediated by a Single Direction (2024).