Qwen3.6-35B-A3B-abliterated-NVFP4-MTP
The fastest abliterated Qwen3.6 for NVIDIA Blackwell / DGX Spark — NVFP4 quantized with MTP (Multi-Token Prediction) layers for speculative decoding.
95.2 tok/s decode on a single DGX Spark (GB10) with MTP=3.
What Makes This Different
No other model on HuggingFace combines all three:
| Feature | This Model | sakamakismile NVFP4 | RedHatAI NVFP4 |
|---|---|---|---|
| NVFP4 weights | ✅ | ✅ | ✅ |
| MTP layers | ✅ BF16 | ❌ | ✅ |
| Abliterated | ✅ | ✅ | ❌ |
| Speculative decode | ✅ 95 tok/s | ❌ ~70 tok/s | ✅ but censored |
Without MTP layers, you're limited to ~70 tok/s on GB10. MTP enables speculative decoding which gives a +35% speedup.
Model Details
- Architecture: Qwen3 MoE (Mixture of Experts), 35B total / 3B active parameters
- Main weights: NVFP4 quantized (21.9 GB) — optimized for NVIDIA Blackwell CUTLASS kernels
- MTP layers: BF16 (1.6 GB, 19 tensors in
model-mtp.safetensors) - Total size: ~23.5 GB
- Context length: 131,072 tokens
- Abliteration: Refusal direction removed via representation engineering (from huihui-ai)
How It Was Made
- Started with sakamakismile/Huihui-Qwen3.6-35B-A3B-abliterated-NVFP4 (NVFP4 main weights, no MTP)
- Extracted BF16 MTP layers from huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated (full BF16 model)
- Injected MTP tensors into a separate
model-mtp.safetensorsfile - Updated
model.safetensors.index.jsonwith MTP weight mappings - Verified with vLLM speculative decoding (MTP=1 through MTP=4)
Benchmark Results
Hardware: NVIDIA DGX Spark (GB10 Blackwell, sm_121, ~128GB unified memory) Engine: vLLM v0.23.1 with FlashInfer + CUTLASS NVFP4 MoE kernels Benchmark: 256 generated tokens at various context lengths
| Config | 1K ctx | 4K ctx | 16K ctx | 32K ctx | 64K ctx |
|---|---|---|---|---|---|
| MTP=3, seqs=1 (best) | 92.2 | 95.2 | 90.7 | 89.7 | 84.4 |
| MTP=2, seqs=4 | 87.2 | 75.5 | 82.8 | 81.6 | 74.4 |
| MTP=4, seqs=1 | 75.6 | 71.3 | 74.0 | 77.6 | 74.4 |
| No MTP (NVFP4 only) | ~71 | ~70 | ~74 | ~68 | ~70 |
MTP=3 is the sweet spot. MTP=4 regresses due to speculation overhead exceeding acceptance rate.
Recommended vLLM Launch Config
docker run -d --gpus all --network host --ipc host --shm-size=16g \
--name vllm-qwen36 \
-e VLLM_MARLIN_USE_ATOMIC_ADD=1 \
-e TORCH_MATMUL_PRECISION=high \
-v /path/to/models:/models \
ghcr.io/spark-arena/dgx-vllm-eugr-nightly:latest \
vllm serve /models/Qwen3.6-35B-A3B-abliterated-NVFP4-MTP \
--host 0.0.0.0 --port 8000 --max-model-len 131072 \
--max-num-batched-tokens 32768 --max-num-seqs 1 \
--trust-remote-code --gpu-memory-utilization 0.7 \
--reasoning-parser qwen3 --kv-cache-dtype fp8 \
--load-format instanttensor --attention-backend flashinfer \
--speculative-config '{"method": "mtp", "num_speculative_tokens": 3}' \
--enable-prefix-caching -tp 1