Qwen3.8-27B NVFP4 Q5K — Physical no-MTP, 16 GB VRAM
This repository contains a physical no-MTP derivative of a Qwen3.8-27B NVFP4/Q5K GGUF and the matching F16 Vision projector.
Current best 16 GB deployment
Validated on an NVIDIA GeForce RTX 5060 Ti 16 GB on 2026-08-16:
Qwen3.8-27B NVFP4 Q5K
physical no-MTP
66K shared KV
P2 / unified KV
full-GPU text backbone
Q4_0 K/V cache
Qwen3.8 Vision enabled
F16 mmproj on CPU
--image-max-tokens 4096
llama.cpp upstream b10435 / 9e40df63ba151d771d8b247ac4011cf203337e99
NO local FA patch
Recommended command:
llama-server \
-m Qwen3.8-27B-NVFP4-Q5K-no-MTP.gguf \
--mmproj mmproj-Qwen3.8-27B-F16.gguf \
--no-mmproj-offload \
--image-max-tokens 4096 \
-c 66000 \
-np 2 \
--kv-unified \
-ngl 999 \
--flash-attn on \
-ctk q4_0 -ctv q4_0 \
-b 512 -ub 64 \
--threads 7 \
--fit off \
--jinja \
--host 0.0.0.0 \
--port 8001
Measured memory envelope
fresh startup: 15,810 MiB used / 80 MiB free
post first warm-up: 15,816 MiB used / 74 MiB free
later repeated stress: no further VRAM growth observed
The tested GPU reports 16,311 MiB total VRAM through nvidia-smi, so this is an intentionally tight, hardware-specific profile.
Repeated production-like test
Two live logical lanes were exercised concurrently:
MAIN: ~40K prompt + 2,048 generated tokens
Vision: real image request, ~4,042-4,084 prompt tokens
The MAIN request repeatedly completed at approximately:
prompt processing: ~191-194 tok/s under concurrent Vision load
decode: ~19.7-20.3 tok/s under concurrent Vision load
No CUDA OOM was observed. After first-use warm-up, repeated equivalent runs did not continue increasing VRAM.
Why image-max-tokens 4096 is important
Without an image-token cap, dynamic-resolution Vision requests could expand to tens of thousands of prompt tokens. One failing 66K/P2 test reached:
Vision slot: 35,679 tokens
MAIN slot: 30,432 tokens
combined: 66,111 tokens
The server then returned Context size has been exceeded. This was a shared-KV capacity failure, not a CUDA OOM.
With:
--image-max-tokens 4096
the tested image-recognition workload stayed around 4K prompt tokens and CPU-offloaded Vision latency returned to a practical range.
Patch status — experimental only
An earlier local b10435 Flash-Attention transient-scratch patch is still published for reproducibility:
1b0cb7a04a62543a4f27ce8ae6ef7f08cc79bd246dc282af23e4b3439a6c266b patches/b10435-fa-transient-final.patch
It increased startup VRAM headroom substantially in the earlier 64K/P2 tests. However, later repeated P2 + Vision stress showed undesirable stepwise CUDA-pool growth under the patched path. A 68K sequence measured approximately:
15,778 MiB
15,796 MiB (+18 MiB)
15,814 MiB (+18 MiB)
A similar ~18 MiB stepping pattern was observed again after returning to 64K while keeping the patched P2 + Vision path.
By contrast, clean upstream b10435 at 66K/P2 with --image-max-tokens 4096 stabilized after a small first-use warm-up increase.
Therefore the patch is not recommended for production. Use the clean upstream b10435 build for the current best 16 GB profile. See PATCH-NOTES.md for the detailed A/B notes.
Full-GPU placement matters
Measured single-stream decode:
| Configuration | Decode speed |
|---|---|
| 32K / P1 / full GPU | 25.88 tok/s |
| 72K / P1 / full GPU | 25.85 tok/s |
| earlier partial-offload tests | ~19.8–20.3 tok/s |
For this card, keeping the text backbone fully on GPU is substantially more valuable than gaining a little more context through CPU layer offload.
P2 continuous batching
Earlier 64K/P2/upstream tests measured two simultaneous 512-token generations at approximately:
24.35 tok/s
24.48 tok/s
aggregate ~48.8 tok/s
This established that P2 itself is useful on the card. The final 66K profile adds the Vision token cap while keeping upstream/no-patch behavior.
Vision projector
Matching projector:
mmproj-Qwen3.8-27B-F16.gguf
927.6 MiB
334 tensors
Use:
--no-mmproj-offload
to keep the projector in system RAM.
A separate 72K/P1/upstream/no-patch test showed repeated Vision requests were VRAM-stable after a small first-use warm-up increase.
Physical no-MTP rewrite
Base model:
Qwen/Qwen3.8-27B
The source GGUF contained one embedded MTP / NextN layer. The physical rewrite changed:
| Field | Original | no-MTP |
|---|---|---|
block_count | 65 | 64 |
nextn_predict_layers | 1 | 0 |
| Highest remaining block | 64 | 63 |
| Removed MTP tensors | 15 | — |
| Removed physical size | 227.91 MiB | — |
No requantization was performed. Retained tensors were copied unchanged.
Exact artifact SHA256
828c54b45e711a7579abe007aeea46c4fbadb71cacc07545239ffb6efa332e66 Qwen3.8-27B-NVFP4-Q5K-no-MTP.gguf
71101eb61e223e70e58b762c596f5303b63a91aec45fd9cdc5dad5592377f2ee mmproj-Qwen3.8-27B-F16.gguf
Tested software
Ubuntu: 24.04.4 LTS
NVIDIA driver: 610.43.02
CUDA: 13.3
llama.cpp: b10435
commit: 9e40df63ba151d771d8b247ac4011cf203337e99
CUDA target: SM120
Build options:
GGML_CUDA=ON
CMAKE_CUDA_ARCHITECTURES=120
GGML_CUDA_FA_ALL_QUANTS=ON
CMAKE_BUILD_TYPE=Release
Scheduler guidance
-c 66000 -np 2 --kv-unified is one shared KV pool, not two independent 66K contexts.
Recommended mixed-agent policy:
CS lane: ~4K class, non-thinking, short output, Vision allowed
MAIN lane: use the remaining shared KV for long-context work
Additional concurrent jobs should queue or spill to a remote model rather than consume the reserved CS lane.
Practical warning
The measured post-warm-up free VRAM was only about 74 MiB. A different driver, llama.cpp revision, display load, image shape, batch size, or background CUDA process can change the result. Do not run another CUDA-heavy workload such as ComfyUI at the same time.
This is an independent community deployment project and is not an official Qwen, NVIDIA, Hugging Face, or llama.cpp release.