Qwen3-Coder-Next-NVFP4-GB10
NVFP4 quantization of Qwen/Qwen3-Coder-Next for NVIDIA DGX Spark (GB10).
Qwen3-Coder-Next is a 79.7B-parameter MoE coding model (512 experts, 10 active per token) with hybrid DeltaNet+attention architecture. This quantization uses a GB10-tuned ignore list that quantizes more aggressively than standard NVFP4 configurations.
Model Details
| Base Model | Qwen/Qwen3-Coder-Next |
| Architecture | Qwen3NextForCausalLM (Hybrid MoE — DeltaNet + attention) |
| Total Parameters | 79.7B |
| Active Parameters | ~3B per token (512 experts, 10 active) |
| Quantization | NVFP4 (4-bit floating point) via LLM Compressor |
| Format | compressed-tensors (safetensors), 10 shards |
| Size on Disk | 45.9 GB |
| Context Length | 262,144 tokens (262K) |
| License | Apache 2.0 |
Quantization Details
- Method: Post-training quantization via LLM Compressor
- Calibration Dataset: HuggingFaceH4/ultrachat_200k (train_sft split)
- Calibration Samples: 64
- Max Sequence Length: 2048 tokens
- Environment:
LLMCOMPRESSOR_MOE_CALIBRATE_ALL_EXPERTS=1
Ignore List (layers kept in BF16)
lm_head
model.embed_tokens
re:.*linear_attn.conv1d
re:.*linear_attn.in_proj_ba
re:.*mlp.gate$
re:.*mlp.shared_expert_gate$
Everything else — including in_proj_qkvz — is quantized to FP4. On GB10's 221 GB/s bandwidth, the bandwidth savings from quantizing these layers outweigh the FP4 kernel dispatch overhead.
Performance (Single NVIDIA DGX Spark — GB10, 128 GB)
Benchmarked with llama-benchy v0.3.3, 3 runs per config.
| PP | TG | Prefill (tok/s) | Decode (tok/s) | TTFT (ms) |
|---|---|---|---|---|
| 512 | 128 | 2,024 | 62.0 | 285 |
| 512 | 256 | 2,528 | 62.1 | 206 |
| 1024 | 128 | 3,261 | 60.6 | 319 |
| 1024 | 256 | 3,350 | 61.8 | 309 |
| 4096 | 128 | 3,987 | 61.1 | 1,031 |
| 4096 | 256 | 3,971 | 61.1 | 1,035 |
| Metric | Value |
|---|---|
| Model memory | 42.7 GiB |
| KV cache | 61.7 GiB (1,346,432 tokens) |
| Concurrent sessions @ 262K | ~5 |
| Concurrent sessions @ 65K | ~20 |
The hybrid DeltaNet+attention architecture means decode speed is constant regardless of context length — DeltaNet layers don't use KV cache.
Running on a Single DGX Spark
Docker image: avarok/dgx-vllm-nvfp4-kernel:v23 (vLLM 0.16.0-rc2, CUDA 13.0, SM 12.1)
Download the model:
huggingface-cli download saricles/Qwen3-Coder-Next-NVFP4-GB10 \
--local-dir /opt/huggingface/models/Qwen3-Coder-Next-NVFP4-GB10
Launch:
docker run -d --name coder-next --gpus all --ipc=host --shm-size 32g \
-v /opt/huggingface/models/Qwen3-Coder-Next-NVFP4-GB10:/models/Qwen3-Coder-Next-NVFP4-GB10 \
-p 8000:8000 \
-e VLLM_NVFP4_GEMM_BACKEND=marlin \
-e VLLM_TEST_FORCE_FP8_MARLIN=1 \
-e VLLM_USE_FLASHINFER_MOE_FP4=0 \
-e VLLM_MARLIN_USE_ATOMIC_ADD=1 \
-e MODEL=/models/Qwen3-Coder-Next-NVFP4-GB10 \
-e PORT=8000 \
-e MAX_MODEL_LEN=262144 \
-e GPU_MEMORY_UTIL=0.90 \
-e "VLLM_EXTRA_ARGS=--kv-cache-dtype fp8 --attention-backend flashinfer --enable-prefix-caching --enable-chunked-prefill --max-num-batched-tokens 8192 --max-num-seqs 64 --enable-auto-tool-choice --tool-call-parser qwen3_coder" \
avarok/dgx-vllm-nvfp4-kernel:v23
Test it:
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen3-Coder-Next-NVFP4-GB10",
"messages": [{"role": "user", "content": "Write a Python function to find the longest common subsequence"}],
"temperature": 0.7,
"max_tokens": 2048
}'
Notes
- At 42.7 GiB model weight + 0.90 GPU util, you get ~62 GiB for KV cache — enough for 5 concurrent 262K sessions.
gpu_memory_utilization=0.93works but leaves very little system headroom. 0.90 is safer.- Decode speed is constant across context lengths thanks to the DeltaNet hybrid architecture.
- Marlin backend is 15% faster than VLLM_CUTLASS for this model's 512 experts.
Target Hardware
Quantized and tested on NVIDIA DGX Spark (GB10, 128 GB unified memory, 221 GB/s bandwidth). Should work on other Blackwell GPUs with NVFP4 support.
Acknowledgments
- Base model by Qwen
- Quantization tooling by vLLM / LLM Compressor