Qwen3.6-35B-A3B-NVFP4 (GGUF)

GGUF conversion of RedHatAI/Qwen3.6-35B-A3B-NVFP4 for use with llama.cpp and compatible runtimes.

Model Summary

Architecture

Qwen3.6-35B-A3B is a mixture-of-experts multimodal model with:

• Hybrid attention: alternating linear (Mamba-style SSM) and full attention layers every 4 blocks
• 256 experts with 8 activated per token (grouped-query MoE)
• NVFP4 quantization via compressed-tensors on all Linear weight layers, preserving auxiliary tensors (norms, SSM convolutions, router gates) in BF16/F32
• Vision encoder: Qwen2.5-VL-style ViT with 1152 hidden dim, merged into the language model

How This GGUF Was Created

Prerequisites

1. Clone llama.cpp with NVFP4 support:

   git clone https://github.com/ggml-org/llama.cpp.git
   cd llama.cpp
   git checkout b8967
   

   Version b8967 (or later) includes PR #21095 for converting HuggingFace NVFP4 models quantized with compressed-tensors, plus native Blackwell NVFP4 CUDA support via PR #22196. The converter maps compressed-tensors tensor names (e.g., weight_global_scale) to the ModelOpt equivalents and adjusts scale values to their reciprocals — weights themselves need no adaptation.

2. Install conversion dependencies (Python 3.12 or 3.13):

   pip install -r requirements/requirements-convert_hf_to_gguf.txt --no-cache-dir
   

Conversion Steps

1. Download the source model:

   huggingface-cli download RedHatAI/Qwen3.6-35B-A3B-NVFP4 --local-dir ./model
   

2. Apply tokenizer compatibility patch — the upstream tokenizer_config.json uses TokenizersBackend which is not recognized by the converter:

   sed -i 's/"TokenizersBackend"/"Qwen2Tokenizer"/g' ./model/tokenizer_config.json
   

3. Register the BPE pre-tokenizer hash — the Qwen3.6 tokenizer produces an unrecognized hash in get_vocab_base_pre(). Add this entry to convert_hf_to_gguf.py:

   if chkhsh == "1444df51289cfa8063b96f0e62b1125440111bc79a52003ea14b6eac7016fd5f":
       # ref: https://huggingface.co/RedHatAI/Qwen3.6-35B-A3B-NVFP4
       res = "qwen35"
   

4. Run the conversion:

   python convert_hf_to_gguf.py ./model \
     --outfile Qwen3.6-35B-A3B-NVFP4.gguf \
     --outtype bf16 \
     --verbose
   

Important Notes

• The --outtype bf16 flag does not produce a standard BF16 model. NVFP4-compatible tensors are repacked into the GGUF where supported by the converter and runtime. Auxiliary tensors (layer norms, SSM parameters, router gates) remain in floating-point format.
• The BPE pre-tokenizer hash (1444df51...) was added to get_vocab_base_pre() mapping to the qwen35 tokenizer preset, matching the \p{M} (combining marks) regex used by Qwen3 models.

Benchmark Results

Benchmarks run using llama-bench from the llama.cpp Docker image (fc2b0053f / 8967) on CUDA 13 with a single GPU. Each model was tested at context lengths 2048, 4096, and 8192 tokens (prompt processing and 32-token token generation).

Qwen3.6-35B-A3B: NVFP4 vs Q4_K_XL

Qwen3.6-27B: NVFP4 vs Q4_K_XL

Conclusions

• Prompt processing (PP): NVFP4 is significantly faster than Q4_K_XL across both model sizes. For 35B-A3B: ~7% at 2048 context (+8% at 4096, +7.5% at 8192). For 27B: ~39% faster consistently (~37-40% across contexts). This reflects NVFP4's lower computational overhead during attention computation since weights stay in their native FP4 format rather than being dequantized on-the-fly.
• Token generation (TG): Q4_K_XL is ~28% faster at TG for the 35B-A3B (188 vs 136 t/s). For the 27B the gap is negligible — Q4_K_XL is only ~5% faster (66 vs 63 t/s). The NVFP4 path has higher per-token overhead in MoE routing and FFN computation, but this is less impactful at smaller scale.
• Model size: For 35B-A3B, NVFP4 is marginally larger (20.94 vs 20.81 GB). For 27B, NVFP4 adds ~1.1 GB (17.50 vs 16.39 GB).
• Recommendation: For 27B, NVFP4 is essentially a win — same TG throughput with significantly faster prompt processing. For 35B-A3B the choice depends on workload: NVFP4 for prompt-heavy use cases (RAG with long context windows), Q4_K_XL for interactive chat where per-token latency matters more.

Benchmark Methodology

podman run --rm --name llama-bench-<id> \
  --device=nvidia.com/gpu=all \
  --security-opt label=disable \
  -v ~/.cache/huggingface/:/root/.cache/huggingface \
  ghcr.io/ggml-org/llama.cpp:full-cuda13-b8967 \
  --bench -hf <model> -n 3 --n-prompt <context> --n-gpu-layers 999

Full benchmark script: benchmark.sh

Multimodal Projector

This model includes a bundled multimodal projector (mmproj-BF16.gguf) for vision tasks. It is shared with the base Qwen3.6-35B-A3B architecture and sourced from Unsloth's GGUF conversion.

When using --hf, llama.cpp loads it automatically. For local usage, specify it with:

./llama-cli -m Qwen3.6-35B-A3B-NVFP4.gguf --mmproj mmproj-BF16.gguf -n 256 --prompt "Hello"

Usage

# Local usage (specify mmproj for vision)
./llama-cli -m Qwen3.6-35B-A3B-NVFP4.gguf --mmproj mmproj-BF16.gguf -n 256 --prompt "Hello"

# Via HuggingFace (auto-discovers mmproj)
./llama-cli --hf knoopx/Qwen3.6-35B-A3B-NVFP4-GGUF -n 256 --prompt "Hello"

License

The original Qwen3.6 model is licensed under the Apache 2.0 License. This GGUF conversion is a format transformation of that checkpoint and inherits the same license. See RedHatAI/Qwen3.6-35B-A3B-NVFP4 for full licensing details.