Qwen3.8-27B-NVFP4-BUDGET-GGUF

Two compact GGUF files of Qwen3.8-27B (the native vision-language 27B dense model, Gated DeltaNet + Gated Attention hybrid layout, 262,144-token native context), converted from unsloth/Qwen3.8-27B-NVFP4 — without the MTP speculative head.

These are the budget members of the esatapedico/Qwen3.8-27B-NVFP4-MTP-GGUF family: they share the same byte-identical 448-tensor NVFP4 backbone (all 64 layers' attention + MLP, 13.69 GB) as the compact MTP tiers, but the MTP draft block has been stripped and the extra tensors pinned to the smallest types. No drafter, no speculative decoding — just the model at the lowest possible footprint.

The goal here is to let 16 GB VRAM users benefit from native NVFP4. These are the smallest possible ways to serve this model on a single 16 GB Blackwell card. There are, of course, accuracy trade-offs vs the larger tiers, plus the lack of MTP — like every compact tier of the family, the attention path is re-quantized to NVFP4 (a second quantization step from the source's F8), and here the head tensors are pinned to the smallest types. You can also fit Qwen3.8-27B in a 16GB VRAM GPU through non-NVFP4 variants instead - e.g. those from Unsloth's own original GGUF repository (unsloth/Qwen3.8-27B-GGUF).

• BUDGET — the quality-leaning budget tier: Q3_K lm_head + Q2_K token embedding. Single-16GB friendly.
• STARVED — the absolute floor: Q2_K lm_head and Q2_K token embedding. Smallest possible variant of the family.

Follow along & support

I post updates on new conversions, benchmarks, and what I'm working on over on Ko-fi. Follow along there to keep up with new releases and the work in progress. If you'd like to support more of it, a coffee is always welcome. I do this on consumer hardware and like seeing how far it goes. More is on the way.

☕ ko-fi.com/esatapedico. Updates, work-in-progress, and an optional coffee.

The two files

Both contain 1,187 tensors: the 448-tensor NVFP4 backbone (byte-identical across the whole family, verified per-tensor SHA-256) + 737 F32 norms/scales/gates + the two head tensors. qwen35.nextn_predict_layers=0 and qwen35.block_count=64 (the MTP block is gone), so they load in any llama.cpp build without the draft-mtp path.

Why these exist

The MTP family is built around a byte-identical NVFP4 backbone; the MTP draft head costs VRAM and compute on every pass. If you don't use speculative decoding, the stripped versions free that head entirely and shave the extra tensors down to the cheapest types — the smallest possible way to serve this model on a single 16 GB Blackwell card.

Vision works. The model is a native VLM (images and video). Pair either GGUF with the mmproj-BF16.gguf from the MTP repo (byte-identical to unsloth/Qwen3.8-27B-GGUF's projector) via --mmproj. If you need the absolute minimum VRAM footprint on a single 16 GB card, you can also load the model without the projector (no --mmproj, i.e. text-only) — that frees the projector's ~1 GB and is another lever to fit bigger context.

Benchmarks (32k context, q4_0 KV, one fresh pod per tier)

Hardware: a single RTX 5070 Ti (16 GB Blackwell). Payload: a 28k-token source document, max_tokens 2000. Single-run, indicative only. On reading the numbers: prefill t/s is an average over the run and degrades as context grows, so these figures are not comparable to numbers from different context sizes, hardware, or harnesses (e.g. the 180k-context observations or the RTX 5090 community benchmark in the MTP repo). Compare only within the same context and setup.

• Both are near-identical on prefill (~2288 t/s) and decode (~27.2 t/s); STARVED is ~1% faster on both.
• Single-16GB VRAM probes (RTX 5070 Ti): BUDGET peaks 15.9 GiB, STARVED 15.7 GiB — both fit one 16 GB Blackwell card.

Usage

llama.cpp / llama-server

llama-server \
  --model Qwen3.8-27B-NVFP4-BUDGET.gguf \
  --mmproj mmproj-BF16.gguf \
  --ctx-size 262144 \
  --flash-attn on \
  --temp 0.7 --top-p 0.95 --top-k 20

• Requires a recent llama.cpp with NVFP4 (GGML type 40) CUDA kernels and sm_120 support (Blackwell).
• No --spec-type draft-mtp — these files have no MTP head.
• For vision input, pass --mmproj mmproj-BF16.gguf (from the MTP repo).
• Qwen3.8's official sampling presets: thinking mode temp 1.0 / top_p 0.95 / top_k 20; instruct mode temp 0.7 / top_p 0.80 / top_k 20. Pick per use case.

Attribution & provenance

This is a derivative work built entirely from existing Apache-2.0 artifacts. Nothing here was trained or fine-tuned. Credit belongs to:

1. Alibaba / Qwen team for the base model, Qwen/Qwen3.8-27B (Apache-2.0).
2. Unsloth for the NVFP4 quantization unsloth/Qwen3.8-27B-NVFP4 (Apache-2.0) and the vision projector unsloth/Qwen3.8-27B-GGUF (Apache-2.0, mmproj-BF16.gguf).
3. This repo's author for the GGUF conversion, the tier splicing, and stripping the MTP head.

The NVFP4 tensors are native GGML type 40, preserved from the source checkpoint through GGUF conversion with no re-quantization round trip.

Repository contents

• Qwen3.8-27B-NVFP4-BUDGET.gguf (14.72 GB)
• Qwen3.8-27B-NVFP4-STARVED.gguf (14.59 GB)

SHA-256

1624edf9876cbb1007ba52b0026e7b7977042781bddeef0a7af7cbee9cd270b7  Qwen3.8-27B-NVFP4-BUDGET.gguf
e580923456bcfd5979173ad828ac4db3444cd3fc06b99956960e7f6bc0c855f6  Qwen3.8-27B-NVFP4-STARVED.gguf

See also

The full MTP family (ORIG + VERY-LOW / LOW / MEDIUM / HIGH / VERY-HIGH / HIGHEST, all with the speculative head baked in) lives in esatapedico/Qwen3.8-27B-NVFP4-MTP-GGUF.

License

Apache-2.0, identical to every upstream artifact. The base model license governs; GGUF conversion and quantization are transformations, not new training. When redistributing, please retain attribution to Qwen (Alibaba) and Unsloth as above.

"Qwen" is a trademark of Alibaba. Trademarks are used here only to identify upstream models; this repository is not affiliated with, sponsored by, or endorsed by Alibaba or Unsloth.

Note on this card

This model card was written by an AI assistant at the request of the repository author, who did the engineering. As with any AI-generated text, there may be errors; please verify anything important (hashes, sizes, commands) against the file itself before relying on it.