Qwen3.6-35B-A3B NVFP4 GGUF
NVFP4 (native Blackwell FP4) GGUF of Qwen3.6-35B-A3B (Mixture-of-Experts, ~3B active parameters per token). A single-file, FP4-native quantization that runs on LocalAI's paged-attention llama.cpp backend with strong decode throughput and a low memory footprint on Blackwell-class GPUs (GB10 / DGX Spark).
Blackwell GPU recommended (native FP4-MMA). It also runs on other hardware via NVFP4 dequant, but slower - off Blackwell the FP4 matmuls fall back (e.g. to CPU on Apple Metal). The throughput figures below are measured on GB10 / DGX Spark (consumer Blackwell).
File
| File | Size | file_type | NVFP4 tensors | sha256 |
|---|---|---|---|---|
q36-35b-a3b-nvfp4.gguf | 23.9 GB (23,850,227,680 bytes) | MOSTLY_NVFP4 | 241 | 1690d0424e232527b8bb135a38033e4699ad11817677eebacd40349020faea52 |
Source / attribution
- Weights: MoE NVFP4 weights from
nvidia/Qwen3.6-35B-A3B-NVFP4. - Conversion: converted to GGUF with llama.cpp using
--outtype auto(NVFP4-preserving). The NVFP4 tensors are carried through into the GGUF unchanged, so the file reportsfile_type = MOSTLY_NVFP4with 241 native NVFP4 tensors. No re-quantization to an integer K-quant is performed.
base_model: nvidia/Qwen3.6-35B-A3B-NVFP4
License
Released under the Qwen license (see https://huggingface.co/Qwen/Qwen3.6-35B-A3B/blob/main/LICENSE). This repository redistributes the converted GGUF with attribution to the upstream weights; the original model license and terms apply.
Install in LocalAI
This model is pre-configured in the LocalAI model gallery for the paged-attention backend - install it by name and it is ready to serve, with no manual configuration.
Gallery model name: qwen3.6-35b-a3b-nvfp4-paged
Heads-up - the backend is not in a release yet. The
llama-cpp-localai-pagedbackend and these gallery entries ship in LocalAI PR #10462, which is not yet merged. Until it lands, run LocalAI built from that PR branch. Once it is merged, a current LocalAI release already includes the backend and this gallery entry - nothing extra needed.
Install from the CLI:
local-ai models install qwen3.6-35b-a3b-nvfp4-paged
Or from the WebUI: open the Models gallery, search qwen3.6-35b-a3b-nvfp4-paged, and click Install.
Then call it through the OpenAI-compatible API:
curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "qwen3.6-35b-a3b-nvfp4-paged",
"messages": [{"role": "user", "content": "Hello!"}]
}'
Pre-configured serving config
The gallery entry already applies the optimized paged setup the benchmarks below use - no tuning required:
- Paged KV cache (on-demand, paged-attention backend)
- Decode-first prefill budget (prefill scheduled to keep decode latency low)
- Flash attention
- Optional bf16-SSM fast mode via the
ssm_bf16_tauoption (see Benchmarks)
LocalAI: https://github.com/mudler/LocalAI
Benchmarks
Measured on GB10 / DGX Spark with the LocalAI paged-attention setup:
- Decode throughput reaches ~86% of vLLM at 128-way concurrency (bit-exact, MoE active-expert path).
- 1.5-3x lower memory than the comparable vLLM deployment.
- At equal-or-higher precision (NVFP4 native execution).
- The opt-in bf16-SSM fast mode (
ssm_bf16_tau) closes most of the remaining gap.
Verification
sha256sum q36-35b-a3b-nvfp4.gguf
# 1690d0424e232527b8bb135a38033e4699ad11817677eebacd40349020faea52
SHA256SUMS.txt (covering both NVFP4 GGUFs in this collection) is included in the repo.