Qwen3.5-122B-A10B-abliterix-NVFP4

NVFP4 (W4A4) quantization of wangzhang/Qwen3.5-122B-A10B-abliterix — the 122B/A10B Abliterix-abliterated (uncensored) Qwen3.5 MoE — packaged for vLLM serving on NVIDIA Blackwell hardware (DGX Spark GB10 / SM121).

Summary

Why this quantization

The Abliterix-trained uncensored behavior (0.5% refusal rate, KL divergence 0.0115 vs the Qwen3.5-122B-A10B baseline) is preserved while dropping weight memory from BF16 (~230 GB) to NVFP4 (~70 GB) — fitting comfortably on two DGX Spark nodes (2 × 119 GiB unified memory) with headroom for KV-cache, with the same compressed-tensors runtime path as the FP8 sibling.

Compared to the FP8 W8A8 variant (bjk110/Qwen3.5-122B-A10B-abliterix-FP8, 116 GB), this NVFP4 build cuts another ~40% off-disk and unlocks the FlashInfer-CUTLASS NVFP4 MoE backend for higher throughput on Blackwell.

Quantization method

Direct safetensors-level conversion via convert_bf16_to_nvfp4.py, mirroring the layout of RedHatAI/Qwen3.5-122B-A10B-NVFP4 (built with llm-compressor) — but applied directly to the abliterix BF16 checkpoint, because llm-compressor 0.10 pins transformers <=4.57.6 while Qwen3.5MoeForCausalLM is only available in transformers >=5.5. The direct script needs only torch + safetensors.

For each 2D Linear weight W (shape [out, in]) not in the ignore list:

1. Identify the fused group the Linear belongs to (if any):
   • Attention QKV: q_proj, k_proj, v_proj of the same self_attn → 3-member group
   • Expert gate-up: experts.N.{gate_proj, up_proj} → 2-member group
   • Shared-expert gate-up: shared_expert.{gate_proj, up_proj} → 2-member group
   • Everything else (o_proj, down_proj) is independent.
2. Compute the per-tensor weight_global_scale (float32). For a fused group, all members share one global scale computed from max(abs(W)) across the entire group; vLLM merges these Linears at runtime (fused QKV / gate_up matmul) and requires a single coherent scale per merged matrix — independent per-projection scales produce garbage outputs (repeated tokens) even though the model boots.
3. Reshape W into groups of 16 along the input dim. Compute per-group fp8 e4m3fn weight_scale (one scale per 16-element group) so that each group fits the NVFP4 [-6, 6] range after dequant.
4. Quantize each group element to NVFP4 (e2m1), pack two values per uint8 → weight_packed.
5. The matching input_global_scale (float32, per-tensor) is stored for the runtime to compute on-the-fly NVFP4 activation quantization.

Activations are NVFP4 W4A4 — quantized at inference time by vLLM using the stored input_global_scale with dynamic local group_size=16 (fp8 e4m3fn scale).

config.json quantization block (excerpt)

{
  "quantization_config": {
    "config_groups": {
      "group_0": {
        "format": "nvfp4-pack-quantized",
        "targets": ["Linear"],
        "weights": {
          "num_bits": 4, "type": "float", "strategy": "tensor_group",
          "group_size": 16, "symmetric": true, "dynamic": false,
          "observer": "memoryless_minmax",
          "scale_dtype": "torch.float8_e4m3fn"
        },
        "input_activations": {
          "num_bits": 4, "type": "float", "strategy": "tensor_group",
          "group_size": 16, "symmetric": true, "dynamic": "local",
          "observer": "static_minmax",
          "scale_dtype": "torch.float8_e4m3fn"
        },
        "output_activations": null
      }
    },
    "format": "nvfp4-pack-quantized",
    "ignore": [
      "re:.*lm_head",
      "re:visual.*",
      "re:model.visual.*",
      "re:.*mlp.gate$",
      "re:.*embed_tokens$",
      "re:.*shared_expert_gate$",
      "re:.*linear_attn.*"
    ],
    "quant_method": "compressed-tensors"
  }
}

Serving with vLLM

# Tested with the v022-d568 image — see DGX Spark notes below.
vllm serve bjk110/Qwen3.5-122B-A10B-abliterix-NVFP4 \
    --tensor-parallel-size 2 \
    --distributed-executor-backend ray \
    --quantization compressed-tensors \
    --moe-backend flashinfer_cutlass \
    --kv-cache-dtype fp8 \
    --max-model-len 32768 \
    --max-num-seqs 4 \
    --gpu-memory-utilization 0.88 \
    --enable-chunked-prefill \
    --reasoning-parser qwen3

--moe-backend flashinfer_cutlass selects the FlashInfer CUTLASS NVFP4 grouped-GEMM kernel for the MoE expert path — required to get the full NVFP4 speedup on Blackwell.

A single Blackwell-class GPU is not enough to hold the 70 GB weights + KV cache + activation buffers, so --tensor-parallel-size 2 (or higher) is mandatory.

DGX Spark (GB10, SM121) notes

NVIDIA DGX Spark uses SM121, which on stock vLLM v0.21.0 was excluded from the Marlin/CUTLASS NVFP4 codepaths (the gates were SM120-only). vLLM PR #35568 (commit 06d020bb6) widens those gates to the SM12x family — covering both the FP8 W8A8 and the NVFP4 dispatch paths.

With that fix applied, the boot log reports the NVFP4 kernels are bound to the SM121 GPUs and the FlashInfer-CUTLASS MoE path is selected, confirming the full Blackwell NVFP4 pipeline is active.

Runtime stack (image v022-d568)

The image is the cumulative top of the v022 forward-stack build chain, rooted in NGC nvcr.io/nvidia/pytorch:26.04-py3 (CUDA 13.0, PyTorch 2.12.0a0). Each layer corresponds to one published image tag:

Building on NGC 26.04 (vs. the older 26.03 base used by v021) gives the SM121 GPU the matching CUDA 13.0 driver/runtime split that the Blackwell NVFP4 kernels expect, and is required for FlashInfer v0.6.11.post3 (which assumes CUDA 13 headers).

Lineage

Citation

@software{abliterix,
  author = {Wu, Wangzhang},
  title  = {Abliterix: Automated LLM Abliteration},
  year   = {2026},
  url    = {https://github.com/wuwangzhang1216/abliterix}
}

Acknowledgements

• Wu Wangzhang for the Abliterix framework and the BF16 abliterated checkpoint.
• Qwen team for the Qwen3.5-122B-A10B base model.
• Red Hat AI for the Qwen3.5-122B-A10B-NVFP4 reference layout.
• vLLM compressed-tensors + FlashInfer-CUTLASS NVFP4 MoE integration teams.
• DGX Spark SM121 enablement: vLLM PR #35568 by Blake Ledden (Second Nature Computing) + contributors.