Qwen 3.6 35B A3B NVFP4 Fast GGUF
About
This model is an unmodified GGUF quantization of unsloth/Qwen3.6-35B-A3B-NVFP4-Fast, made with the llama.cpp conversion tool. Please refer to unsloth repository for quality metrics. Your feedback, contributions, optimization proposals and questions are highly appreciated.
This GGUF is primarily intended to be run fully on the GPU without offloading to the CPU. However, you can absolutely run it with offloading. The relevant guides are provided in this repo.
Speed metrics
I've tested this model on my setup (5070 Ti 16GB, 5060 Ti 16GB, 32GB DDR5). Token generation and prefill speeds will vary depending on your hardware. Please use the metrics below as a rough estimate of your potential performance.
| Setup | Token generation, t/s | Prompt prefill, t/s |
|---|---|---|
| 5070ti + 5060ti | ~134.4 | ~3462 |
| 5070ti + CPU | ~68.7 | ~324 |
| 5060ti + CPU | ~51.0 | ~161 |
Quick start
This GGUF requires a recent build of llama.cpp. If the commands below raise a compatibility error, please update your llama.cpp version. All commands have been tested on llama.cpp version b9957 - ensure your version is >=b9957.
Agentic tasks
docker run --rm \
--runtime nvidia \
--gpus 2 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-e HF_HUB_CACHE=/root/.cache/huggingface/hub \
-p 8080:8080 \
ghcr.io/ggml-org/llama.cpp:server-cuda13 \
-hf catlilface/Qwen3.6-35B-A3B-NVFP4-Fast-GGUF \
-np 1 \
-ngl -1 \
-c 256000 \
--no-op-offload \
--spec-type draft-mtp \
--spec-draft-n-max 3
Speed-efficient
The token generation speeds listed below were achieved using varying prompt lengths, KV cache quantization, and the more powerful GPU 0. These optimizations simply let more computations to be done on 5070ti.
docker run --rm \
--runtime nvidia \
--gpus 2 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-e HF_HUB_CACHE=/root/.cache/huggingface/hub \
-p 8080:8080 \
ghcr.io/ggml-org/llama.cpp:server-cuda13 \
-hf catlilface/Qwen3.6-35B-A3B-NVFP4-Fast-GGUF \
-np 1 \
-ngl -1 \
-c 16000 \
-ctk q8_0 \
-ctv q8_0 \
--tensor-split 2,1 \ # only for unequal gpus
--no-op-offload \
--spec-type draft-mtp \
--spec-draft-n-max 3