Qwen3.6-27B NVIDIA NVFP4 no-MTP for 16GB GPUs
Reproducible scripts and deployment notes for a no-MTP derivative of utautako/Qwen3.6-27B-NVIDIA-NVFP4-MTP-GGUF.
The single embedded MTP / next-token-prediction layer is physically removed with llama-quantize in COPY mode. The remaining model tensors and quantization are preserved. This saves 263,314,336 bytes (251.12 MiB) and made a verified full-GPU, 65,536-token deployment possible on one RTX 5060 Ti 16GB using quantized KV cache.
Provenance and scope
- Base model:
Qwen/Qwen3.6-27B - NVIDIA checkpoint:
nvidia/Qwen3.6-27B-NVFP4 - GGUF source conversion:
utautako/Qwen3.6-27B-NVIDIA-NVFP4-MTP-GGUF - Modification and RTX 5060 Ti validation: Wilson Zhang /
wilsonzhang2
This is not a fresh NVFP4 quantization and is not an official Qwen, NVIDIA, utautako, or llama.cpp release. It is a modified GGUF derivative optimized for a tight 16GB VRAM budget.
What changed
The source GGUF declares:
qwen35.block_count = 65
qwen35.nextn_predict_layers = 1
The build script:
- prunes layer
64, the embedded MTP layer; - sets
qwen35.block_countto64; - sets
qwen35.nextn_predict_layersto0; - copies all retained tensors without requantizing them.
Expected output:
File: Qwen3.6-27B-NVIDIA-NVFP4-no-MTP.gguf
Size: 15,484,336,608 bytes (14.4209 GiB)
SHA256: 8e319bca3cd808c2cb61e616a124390979c5f6732b293217b4b89fd826b72d7a
Verified configuration
| Item | Verified value |
|---|---|
| GPU | NVIDIA GeForce RTX 5060 Ti 16GB |
| Driver | 580.173.02 |
| llama.cpp | build 10023, commit b3c9d1b84 |
| Context | 65,536 tokens |
| GPU offload | all layers |
| KV cache | q4_0 K + q4_0 V |
| Parallel slots | 1 |
| Batch / micro-batch | 512 / 64 |
| Observed VRAM after load | approximately 15,814 MiB of 16,311 MiB |
| Short non-thinking generation | approximately 24.9 tok/s; single local test, not a standardized benchmark |
| Controlled TTFT benchmark | not yet published |
The test establishes that the configuration loads and serves requests. It does not establish identical behavior on every 16GB card, driver, llama.cpp revision, display-attached GPU, or operating system.
Build the derivative
Requirements:
- the upstream MTP GGUF;
- a recent
llama-quantizesupporting--prune-layersand--override-kv; - enough free disk space for the source, temporary output, and final output;
- sufficient RAM or swap for the copy operation.
scripts/build-no-mtp.sh \
/path/to/Qwen3.6-27B-NVIDIA-NVFP4-MTP.gguf \
/path/to/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP.gguf
The script refuses to overwrite an existing destination and validates the expected output size and SHA256.
Run at 64K on one 16GB GPU
MODEL=/path/to/Qwen3.6-27B-NVIDIA-NVFP4-no-MTP.gguf \
LLAMA_SERVER=/path/to/llama-server \
scripts/run-64k.sh
Equivalent core command:
llama-server \
--model Qwen3.6-27B-NVIDIA-NVFP4-no-MTP.gguf \
--ctx-size 65536 \
--n-gpu-layers all \
--split-mode none \
--flash-attn on \
--cache-type-k q4_0 \
--cache-type-v q4_0 \
--batch-size 512 \
--ubatch-size 64 \
--parallel 1 \
--threads 7 \
--fit off \
--spec-type none \
--jinja
A sample systemd unit is included at systemd/qwen27b.service.
Why no MTP?
The upstream GGUF includes one MTP layer. Merely disabling speculative decoding does not necessarily remove that layer from the model file or VRAM budget. The physical removal saves about 251 MiB. On the verified RTX 5060 Ti configuration, that margin was the difference between a practical 64K full-GPU deployment and an over-constrained setup.
On GPUs with more VRAM, the original MTP build may be faster and is generally preferable when its draft layer and draft KV cache fit without forcing CPU offload.
Limitations
- Text inference was tested; multimodal inference and an
mmprojwere not validated here. - No standardized quality regression suite has been run specifically against the no-MTP derivative. Removing a speculative draft layer should not change the retained main-model logits, but users should validate their workloads.
- 64K allocation is extremely tight. Desktop display usage, another CUDA process, larger batches, multiple slots, or different compute buffers can cause OOM.
- The published performance number is a local observation, not a cross-platform benchmark.
License and attribution
The upstream model and conversion repositories identify Apache-2.0 as their license. This repository includes the Apache License 2.0 text and an attribution notice. Preserve upstream model-card terms and attribution when redistributing the GGUF.
The scripts and documentation authored in this repository are also released under Apache-2.0. See LICENSE, NOTICE, and ATTRIBUTION.md.