Qwen3.5-35B-A3B NVFP4 gguf

qwen 3.5 quant with experts quantized to nvfp4. It is still slow but shows to be having a better accuracy than mxfp4. This makes sense because both Q4_K and nvfp4 use a superblock scale. I hope that since its a proprietary format from nvidia that it will get better support and be faster than int4. Quantized from BF16, no imatrix used. Used llama.cpp version 3a14a542f5ce8666713c6e6ea44f7f3e01dd6e45 to quantize and calculate kld.

Quant config:

token_embd=Q8_0
attn_gate=Q8_0
attn_norm=F32
attn_qkv=Q8_0
ffn_down_exps=NVFP4
ffn_down_shexp=Q8_0
ffn_gate_exps=NVFP4
ffn_gate_inp=F32
ffn_gate_inp_shexp=Q8_0
ffn_gate_shexp=Q8_0
ffn_up_exps=NVFP4
ffn_up_shexp=Q8_0
post_attention_norm=F32
ssm_a=F32
ssm_alpha=Q8_0
ssm_beta=Q8_0
ssm_conv1d=F32
ssm_dt.bias=F32
ssm_norm=F32
ssm_out=Q8_0
attn_k=Q8_0
attn_k_norm=F32
attn_q=Q8_0
attn_q_norm=F32
attn_v=Q8_0
attn_output=Q8_0
output=Q8_0
output_norm=F32

kld data

UPDATE 20260420: https://github.com/ggml-org/llama.cpp/discussions/22042 It looks like the per tensor scale is not implemented yet. Hopefully this will be implemented soon as we can expect a significant KLD improvement from that. Excited to see this!

Final estimate: PPL = 6.8908 +/- 0.04680

llama-perplexity results

I did not use an imatrix because nvfp4 should not be affected by it.

performance

On 2x 5060ti

CUDA : ARCHS = 1200 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | FA_ALL_QUANTS = 1 | BLACKWELL_NATIVE_FP4 = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |

nvfp4 (update 20260403)

performance improved slightly making this usable.

nvfp4 (update 20260420)

I tested michaels branch here: https://github.com/ggml-org/llama.cpp/pull/21896 So it is getting closer performance wise.

mxfp4

q4_k