Huihui-gemma-4-31B-it-abliterated-v2-NVFP4

This repository contains an NVFP4-compressed version of huihui-ai/Huihui-gemma-4-31B-it-abliterated-v2.

The goal of this build is to produce an NVFP4 checkpoint that actually fits on a single 32 GB RTX 5090 while preserving the multimodal pipeline and the abliteration intervention. The existing NVIDIA reference release keeps every self_attn layer in BF16, which pushes the on-disk weights past 32 GB and makes single-GPU serving impossible. This build quantizes self_attn as well, so only the vision tower, embeddings, and lm_head remain in BF16.

What Changed In v2 (vs v1)

The source model (huihui-ai/...-abliterated-v2) differs from v1 in one key way: the first 5 text layers (layers 0–4) are no longer abliterated. They retain the original google/gemma-4-31B-it weights. Layers 5–59 are still abliterated but with a refusal direction recomputed excluding those early layers.

Per the source model card, this produces lower perplexity (better quality) while maintaining the same level of refusal removal:

In local side-by-side testing of the NVFP4 quantized versions, v2 showed tighter instruction following (stricter JSON output, more efficient reasoning steps within the same token budget) while retaining identical abliteration effectiveness and decode throughput (~69 tok/s).

Source And References

• Source model: huihui-ai/Huihui-gemma-4-31B-it-abliterated-v2
• Upstream base model: google/gemma-4-31B-it
• NVIDIA reference release (conservative ignore list): nvidia/Gemma-4-31B-IT-NVFP4
• RedHatAI reference release: RedHatAI/gemma-4-31B-it-NVFP4
• Quantization library: vllm-project/llm-compressor (main branch, examples/multimodal_vision/gemma4_example.py as starting template)
• vLLM build used for validation: vllm/vllm-openai:gemma4-cu130

Supported Modalities

This checkpoint matches the input/output capabilities of the upstream google/gemma-4-31B-it exactly — no modality was added or removed by quantization.

A Note On The any-to-any Tag Seen On Some Related Repos

The huihui source repository carries an any-to-any tag among its Hub tags. That tag does not reflect the actual capabilities of the 31B weights — the upstream google/gemma-4-31B-it is tagged as image-text-to-text and its model card explicitly restricts audio support to the E2B/E4B variants. The Gemma4Processor in processor_config.json does include a Gemma4AudioFeatureExtractor section (because the Gemma4ForConditionalGeneration architecture class is a superset capable of hosting audio), but the 31B checkpoint ships without audio encoder weights. This release intentionally does not carry an any-to-any tag, to avoid propagating that mislabel.

What Was Quantized

• Every Linear layer in the text stack — including self_attn.{q,k,v,o}_proj and all mlp.*_proj — was quantized to NVFP4 with llm-compressor.
• The vision tower, all embed_* tensors, the lm_head, and any audio_* modules remain in BF16 so the multimodal pipeline and vocabulary pathways are preserved.
• Calibration used mit-han-lab/pile-val-backup, 128 samples at 1024 tokens per sample, batch size 1.
• Quantization recipe (from recipe.yaml):

  QuantizationModifier:
    targets: [Linear]
    ignore: ['re:.*vision.*', 're:.*audio.*', lm_head, 're:.*embed.*']
    scheme: NVFP4
  

Ignore List Compared To The NVIDIA Reference

Inference

As tested locally, this model works with the vllm/vllm-openai:gemma4-cu130 image on an RTX 5090 (32 GB). Representative launch via docker compose:

services:
  vllm:
    image: vllm/vllm-openai:gemma4-cu130
    network_mode: host
    ipc: host
    devices:
      - nvidia.com/gpu=all
    volumes:
      - ./Huihui-gemma-4-31B-it-abliterated-v2-NVFP4:/models/huihui-gemma4-nvfp4:ro
    environment:
      - VLLM_NVFP4_GEMM_BACKEND=marlin
      - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
    command:
      - /models/huihui-gemma4-nvfp4
      - --served-model-name=huihui-gemma4-31b-nvfp4
      - --max-model-len=102400
      - --gpu-memory-utilization=0.95
      - --kv-cache-dtype=fp8
      - --trust-remote-code
      - --enable-prefix-caching
      - --enable-auto-tool-choice
      - --tool-call-parser=gemma4

Observed on RTX 5090 at max-model-len=102400, gpu-memory-utilization=0.95:

• ~29.7 GB VRAM occupied (weights + KV cache + CUDA runtime)
• ~69 tokens/second single-stream decode throughput
• Available KV cache: 8.55 GiB, maximum concurrency for 102,400 tokens: 1.15x
• Engine init (profile + KV cache + warmup): ~51 seconds
• Tool calling enabled with native gemma4 parser

Validation

A local quality sweep was run against the served OpenAI-compatible endpoint. All tests passed.

Text (7 / 7)

• English coherence with sentence-count constraint
• Chinese generation with first-sentence geographic-location constraint
• Multi-step arithmetic (two trains converging, correct reasoning chain)
• Strict JSON output — parsed cleanly by json.loads, output was bare JSON without markdown fencing
• 400-word long-form narrative generation — 21 / 21 unique sentences, no degeneration
• Python code generation with assert-based test cases
• Abliteration smoke test: the refusal-vector intervention from the source model is preserved through the NVFP4 round-trip

Image (4 / 4)

Posted real JPEGs via the image_url content part:

Video (1 / 1)

Posted a real 5-second MP4 (~2.85 MB) via the video_url content part. prompt_tokens = 2436, consistent with 32 frames × 70 soft-tokens per frame plus the text wrapper.

Model output: "A white truck and several cars are parked along a road bordering a lush, green park. The scene features tall trees, a grassy area with a path, and a park bench."

Spatial and object-level description are correct. End-to-end latency was ~2 seconds.

v1 vs v2 Comparison (NVFP4)

Both versions were tested with the same quality battery on the same hardware:

Formal Benchmarks

Intentionally omitted for now. They will be added later once a reproducible evaluation harness run is available.

Notes

• Architecture: Gemma4ForConditionalGeneration
• Model type: gemma4
• Pipeline tag: image-text-to-text
• Quantization format: nvfp4-pack-quantized (compressed-tensors)
• KV cache precision at inference time (recommended): fp8
• Requires transformers >= 5.5.0 to load gemma4 configurations
• The multimodal processing files (processor_config.json, chat_template.jinja) are preserved so vision input still works; text-only usage also works unchanged