Gemma 4 31B IT — PrismaQuant 6-bit, abliterated in format

The quantization in this checkpoint is not ours. Format, allocator decision, codebooks and the entire 6-bit build are the work of Robert Tand (rdtand), author of PrismaQuant and of GridBook. This repository modifies the contents of 118 weight planes inside his build and changes nothing else. Please credit him for the quantization and treat this repo as a derivative of rdtand/Gemma4-31B-IT-PrismaQuant-6bit-vllm.

This is an abliterated derivative of rdtand/Gemma4-31B-IT-PrismaQuant-6bit-vllm that was produced without requantizing the model.

The refusal behaviour of the instruction-tuned base was reduced by transplanting weights from an abliterated BF16 donor into the existing quantized checkpoint and re-encoding them on the original quantization grid: same format per tensor, same bit budget, same codebooks, same scales, same renderer. The only thing that changes is the content of the quantized weight planes of 118 tensors.

The practical consequence is that this checkpoint loads, allocates, and serves exactly like its parent — same shard sizes, same byte layout, same KV budget, same throughput class — because nothing about the quantization was re-decided.

Why build it this way

The conventional route is: abliterate in BF16, then quantize the result. That produces a checkpoint in the same format family, but not the same model. For this 31B model at an identical 6.000 bits-per-weight target, re-running PrismaQuant's allocator on the abliterated weights chose a different per-tensor format assignment than the deployed build:

Same budget, different model. Every downstream property that was validated against the deployed build — tensor-level fidelity, memory footprint, kernel path, KV sizing — has to be re-established from scratch.

A second, more specific motivation: we wanted a checkpoint that is provably the deployed model, abliterated rather than a new model that happens to be abliterated, so that a regression after the switch could only be attributed to the weight edit and not to a changed quantization layout.

It is worth stating clearly what we did not find, because it is the common assumption. We tested whether quantization destroys abliteration, by measuring the BF16 abliterated donor and the very same donor requantized, over one identical API path with one prompt set and one classifier:

Quantization did not undo the abliteration. Earlier numbers suggesting otherwise turned out to come from three different measurement setups, not from three different models. The in-format method is therefore not a workaround for a quantization problem — it is a way to keep a validated deployment artifact validated.

Method

1. Donor. A BF16 abliteration of google/gemma-4-31B-it produced with Heretic in ARA mode (rank-1 directional ablation, real PEFT adapters, merged to dense BF16).
2. Target selection. The Linears that write into the residual stream: o_proj (attention output) and down_proj (MLP output). These are the modules through which a refusal direction is expressed additively.
3. In-format re-encoding. For each target tensor the production codes are decoded, replaced by the donor weights, and re-encoded with the production encoder against the frozen scale planes — per-tensor format, group size, global scale and activation scale are read from the existing checkpoint and never recomputed.
4. Byte-range write-back. The output is a copy of the parent in which only the data ranges of the edited tensors are overwritten. Shapes and dtypes are untouched, so the safetensors header and every other tensor are bit-identical by construction rather than by intent.

NVFP4 note for anyone reproducing this: in compressed-tensors, weight_global_scale is stored as a reciprocal (448·6/amax). It is divided by, not multiplied with. Getting this backwards yields a model that loads and produces fluent garbage.

Scope of the edit

Of 2025 tensors, 1907 are bit-identical to the parent and 118 changed; none outside the intended pattern. Within the edited matrices, 712,752,623 of 9,672,327,168 codes changed (7.37 %).

The vision tower is carried in BF16 passthrough by the parent build and is untouched here, which is why image understanding is unaffected (measured below).

Verification

The null-edit test is the one that matters for trusting the rest: encoding unmodified weights with the same encoder must reproduce the original bytes exactly. It does. Any difference observed afterwards is therefore the intended weight substitution and not codec drift.

Measurements

All numbers below are from our own harness, not from public benchmarks, and are reported so the delta against the parent is interpretable. Measured on a single NVIDIA GB10 (DGX Spark, sm_121), vLLM with compressed-tensors NVFP4 + FP8, NVFP4 KV cache, under verified idle (zero requests for ten minutes, GPU at 0 %).

Refusal breakdown for this build (100 prompts, no probe errors, mean answer length 256 characters):

Calibration. 36 hard refusals puts this checkpoint at the level of the full BF16 abliteration of the same model (33) — at unchanged format, unchanged bit budget and unchanged per-tensor assignment. That equivalence is the actual result here.

Known limitations

• One tool-calling case regressed, consistently. The suite goes from 5/5 to 4/5: in one scenario the model calls a tool that was not offered (exec_command in place of apply_patch). Re-measured twice under verified idle, identical both times, with determinism=True across all three repeats — so it is reproducibly wrong rather than flaky. If tool calling is your primary workload, measure it before adopting this build.
• This is not an uncensored model. 36 of 100 adversarial prompts are still refused outright and a further 54 draw hedging language. The refusal tendency is substantially reduced, not removed.
• Abliteration costs capability in general. We measured the axes listed above and they hold; we did not measure reasoning benchmarks, code generation, or multilingual coverage beyond German. Absence of a measured regression is not evidence of none.
• Not a vanilla Transformers checkpoint. It requires a vLLM build with compressed-tensors NVFP4 + FP8 support. AutoModelForCausalLM will not load it.
• The quality metrics in the parent card (KL vs BF16, next-token agreement) were not re-measured for this build. The 118 changed tensors move the model away from the BF16 reference by design, so those figures do not carry over.

How this differs from neighbouring approaches

We also attempted to express this specific edit as a LoRA adapter over the quantized base. The plumbing works — vLLM's LoRA path is quantization-agnostic, and a zero adapter is bit-identical — but the edit itself has no usable low-rank structure: the control matrices concentrate more spectral energy than the targets do. A rank-limited approximation of this particular edit is therefore not available; the full-weight route is what carries it.

Usage (vLLM)

vllm serve TechPrototyper/Gemma4-31B-IT-PrismaQuant-6bit-abliterated-vllm \
  --quantization compressed-tensors \
  --trust-remote-code

What the checkpoint actually requires

A vLLM build with compressed-tensors support for the mixed-precision format, covering both schemes this checkpoint declares:

plus 260 ignore entries (vision tower, norms, embeddings, lm_head). On Blackwell this uses the FlashInfer CUTLASS NVFP4 kernels; FP8 needs Hopper or newer. kv_cache_scheme is null, so no KV-cache quantization is required or implied by the checkpoint.

AutoModelForCausalLM will not load this — it is a vLLM-targeted export.

Note on our internal build (not a requirement)

We serve this model on an in-house vLLM tree rather than a release wheel, and it is worth being explicit that none of our local patches are needed to run this checkpoint. We checked: the only one of them that touches the compressed-tensors code at all relaxes validate_kv_cache_scheme to accept num_bits=4, and since this checkpoint declares no kv_cache_scheme, that code path is never reached. Nothing in our tree touches the weight-loading or Linear quantization path.

What our tree adds is two optional things, both unrelated to these weights:

• NVFP4 KV cache on consumer Blackwell (sm120/sm121) via FlashInfer FA2, including a Gemma-4 VO-split two-pass prefill and multimodal prefix-span masking. This is a serving choice that buys KV capacity; it is not part of the checkpoint.
• A DFlash2 speculative-decoding drafter.

Consequence for the numbers above: our measurements were taken with NVFP4 KV cache and an MTP drafter enabled. The 524,903-token KV budget and the 18–19 tok/s figures are therefore our configuration, not what a stock vLLM will reproduce. The quality and refusal measurements do not depend on either feature; the capacity and throughput ones do.

We have not tested this checkpoint on a stock upstream vLLM release. The reasoning above is from reading our own diff against upstream, not from a passing run. If you try it on a release wheel, we would be glad to hear how it went.

Serve without speculative decoding if you intend to read prompt logprobs.

Intended use and safety

This checkpoint has had its refusal tendency deliberately reduced. It is published for research on quantization-preserving weight edits and for use in settings where the operator carries responsibility for content policy at the application layer. It is not suitable as a drop-in replacement in a product that relies on model-level refusals as its safety mechanism.

Use is governed by the Gemma 4 license, including its prohibited-use policy, which applies to this derivative exactly as it applies to the base model.

Attribution

• Base model © Google, under the Gemma 4 license.

• Quantization: Robert Tand — rdtand. The parent build is rdtand/Gemma4-31B-IT-PrismaQuant-6bit-vllm. Everything that makes this checkpoint a 6-bit model is his: the Fisher-weighted mixed-precision allocator, the per-tensor format menu, the act-aware GPTQ and scale-sweep passes, the codebooks, and the compressed-tensors export path. This repository changes the contents of 118 weight planes inside that build and nothing else — no re-allocation, no re-quantization, no new scales.

  His quantization work spans two formats, and it is worth naming both because the method described here applies to both: PrismaQuant ({NVFP4, FP8_E4M3, BF16} per Linear, exported as compressed-tensors) is what this Gemma checkpoint uses; and GridBook (product-vector quantization against codebooks on a hardware grid, cb_codebooks.pqcb) is his codebook format, on which we ran the same in-format procedure for a hybrid Qwen model. Neither format needed modification to make this work — which is the point: the edit stays inside the format the author defined.

• Abliteration tooling: Heretic (ARA mode).

• In-format transplant, verification and measurements: this repository's author.