Qwen3.6-27B NVFP4 GGUF
NVFP4 (native Blackwell FP4) GGUF of Qwen3.6-27B (dense). A single-file, FP4-native quantization that runs on LocalAI's paged-attention llama.cpp backend with high decode throughput and low memory footprint on Blackwell-class GPUs (GB10 / DGX Spark).
Blackwell GPU recommended (native FP4-MMA). It also runs on other hardware via NVFP4 dequant, but slower - off Blackwell the FP4 matmuls fall back (e.g. to CPU on Apple Metal). The throughput figures below are measured on GB10 / DGX Spark (consumer Blackwell).
File
| File | Size | file_type | NVFP4 tensors | sha256 |
|---|---|---|---|---|
q36-27b-nvfp4.gguf | 25.5 GB (25,475,916,000 bytes) | MOSTLY_NVFP4 | 304 | 2fdd857b13cbaa37b913d9566bf0a69443dcdb702e95694ca8d75236710575d4 |
Source / attribution
- Weights: dense NVFP4 weights from
unsloth/Qwen3.6-27B-NVFP4. - Conversion: converted to GGUF with llama.cpp using
--outtype auto(NVFP4-preserving). The NVFP4 tensors are carried through into the GGUF unchanged, so the file reportsfile_type = MOSTLY_NVFP4with 304 native NVFP4 tensors. No re-quantization to an integer K-quant is performed.
base_model: unsloth/Qwen3.6-27B-NVFP4
License
Released under the Qwen license (see https://huggingface.co/Qwen/Qwen3.6-27B/blob/main/LICENSE). This repository redistributes the converted GGUF with attribution to the upstream weights; the original model license and terms apply.
Install in LocalAI
This model is pre-configured in the LocalAI model gallery for the paged-attention backend - install it by name and it is ready to serve, with no manual configuration.
Gallery model name: qwen3.6-27b-nvfp4-paged
Heads-up - the backend is not in a release yet. The
llama-cpp-localai-pagedbackend and these gallery entries ship in LocalAI PR #10462, which is not yet merged. Until it lands, run LocalAI built from that PR branch. Once it is merged, a current LocalAI release already includes the backend and this gallery entry - nothing extra needed.
Install from the CLI:
local-ai models install qwen3.6-27b-nvfp4-paged
Or from the WebUI: open the Models gallery, search qwen3.6-27b-nvfp4-paged, and click Install.
Then call it through the OpenAI-compatible API:
curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "qwen3.6-27b-nvfp4-paged",
"messages": [{"role": "user", "content": "Hello!"}]
}'
Pre-configured serving config
The gallery entry already applies the optimized paged setup the benchmarks below use - no tuning required:
- Paged KV cache (on-demand, paged-attention backend)
- Decode-first prefill budget (prefill scheduled to keep decode latency low)
- Flash attention
- Optional bf16-SSM fast mode via the
ssm_bf16_tauoption (see Benchmarks)
LocalAI: https://github.com/mudler/LocalAI
Benchmarks
Measured on GB10 / DGX Spark with the LocalAI paged-attention setup:
- Decode throughput reaches ~90-117% of vLLM dense decode (and 96.6% at 128-way concurrency, bit-exact).
- 1.5-3x lower memory than the comparable vLLM dense deployment.
- At equal-or-higher precision (NVFP4 native execution).
- The opt-in bf16-SSM fast mode (
ssm_bf16_tau) beats vLLM dense.
Verification
sha256sum q36-27b-nvfp4.gguf
# 2fdd857b13cbaa37b913d9566bf0a69443dcdb702e95694ca8d75236710575d4
SHA256SUMS.txt (covering both NVFP4 GGUFs in this collection) is included in the repo.