Qwen3.8-27B · KQV-Q4 + IQ3 Floor · MTP

English ｜ 中文（点击跳转）

What I wanted was simple: a small model with high intelligence. Qwen3.8 27B is a popular model, the kind of thing that basically just works for local deployment. The catch is VRAM. BF16 wants ~54 GB, and even the quantized GGUFs out there are still large. On a 16 GB card the context budget is simply not enough, and the uniform-quantization builds that do shrink small just aren't smart enough. I've gone through a fair number of other people's quantized builds, and none of them satisfied me.

I've been doing this for a while. Back in the Qwen3.6 era I was already running this exact playbook, with dozens of quantization comparisons, one after another. What I built then was one of the smallest, smartest builds on the market, and in its time you could almost call it a first. All those runs left me with one conviction: with the same idea, a bigger build is not necessarily a better one. Some combinations are genuinely surprising, and the right size paired with the right precision mix tends to give the best result. There are a lot of small builds out there now, and I haven't tested them all. To be honest, mine may not be the best on the market anymore. But my strategy still works: part of the result depends on the base model, and the other part on the tiered-precision strategy itself. So this file is still useful, especially if you work with limited VRAM. This is that strategy on Qwen3.8: mixed-precision quantized with KQV-Q4 + IQ3 Floor, every K/Q/V anchor block stays at Q4_K or above, every norm stays F32, the FFN layers ride the IQ3_XXS floor, and an importance matrix guides where the low end goes. Built-in MTP. On a 16 GB card with a Q4 KV cache, 130k+ context still fits.

What sets it apart: the full 15-tensor MTP head; 130k+ context on 16 GB. All of it in one 10 GB file.

Built on Qwen3.8; thanks to the Qwen team, and to unsloth for the base weights. In short, this file is a mixed-precision quantization of the unsloth/Qwen3.8-27B-GGUF base weights, with the built-in MTP head.

Files

SHA256 of this model: 33442d8a1f49b3ee243034a01826747eed774162c7ffd1811552d23d38377699

Mixed precision: KQV-Q4 + IQ3 Floor

866 tensors, 3.18 BPW. Same recipe as my other KQV builds:

The type histogram is Q4_K 49 · Q6_K 1 · IQ3_S 130 · IQ3_XXS 326 · F32 360, verified per tensor. The IQ3-level tensors were quantized with an importance matrix, while the MTP-head tensors without matrix entries use standard quantization.

Why this beats the plain 10 GB tier

1. Small size, full anchors: a ~10 GB file while every K/Q/V anchor stays at Q4_K and every norm at F32.
2. The MTP head is inside, and not every 10 GB build has one. This one carries the complete 15-tensor head and runs --spec-type draft-mtp at roughly 2-3× on chat-length outputs.
3. 130k+ context on 16 GB. A same-size f16-KV build gives you a fraction of that. And at that context it still has room to run the vision model alongside: --mmproj runs on the CPU, with almost no speed impact.

MTP

• block_count = 65, nextn_predict_layers = 1. In other words, the MTP head is a full decoder block at blk.64 (11 attention/FFN tensors + 4 nextn tensors).
• Head precision: attn_v = Q6_K (the highest precision in the file), the remaining weights = IQ3_S (one step above the floor), all norms F32.
• The draft head rides one step above the floor, with attn_v at the highest precision in the file. That's what the acceptance rate lives on.

Usage

Tested on llama.cpp b10728:

llama-server.exe `
  -m Qwen3.8-27B-UD-KQVQ4-IQ3Floor-10G-MTP.gguf `
  -c 131072 -ngl 99 -t 8 --kv-unified `
  --jinja --reasoning on --reasoning-format deepseek `
  --spec-type draft-mtp -np 1 --spec-draft-n-max 2 --spec-draft-p-min 0 `
  --cache-type-k q4_0 --cache-type-v q4_0 -fa on `
  --top-k 20 --top-p 0.95 --repeat-penalty 1.1 --min-p 0

• --spec-type draft-mtp: turns on the built-in MTP head (expect ~2-3× on chat-length outputs).
• --cache-type-k q4_0 --cache-type-v q4_0: Q4 KV cache. This is what lets 130k fit in 16 GB.
• For vision, pair it with the Qwen3.8-27B vision projector via --mmproj. It runs on the CPU at almost no speed cost, so it fits even at 130k context.
• LM Studio: import the GGUF directly and enable speculative decoding (MTP) in the server settings.

Verification

Disclaimer

This model is provided AS IS, without warranty of any kind, express or implied; the Apache 2.0 license terms apply. It is a personal research build. Test it in your own environment before relying on it; all risk is yours.

Base model: unsloth/Qwen3.8-27B-GGUF.

中文说明

我想要的一直很简单：一个小体积、高智能的模型。Qwen3.8 27B 是一个很受欢迎的模型，本地部署基本拿来就能用。麻烦在显存上：BF16 要 ~54 GB；而市面上的 GGUF，即使量化了还是很大，对于 16 GB 显存的显卡，上下文还是不够；那些均匀量化压小了的，体积是够小了，智力又不行。我也看过不少别人量化的版本，始终没有一款令我满意的。

这事我做了挺长时间。早在 Qwen3.6 那会儿，我就是这套流程：几十次量化对比，一轮一轮跑下来。当时做出来的，是市场上最小体积与智能兼得的最佳构建之一，当时几乎称得上首创。那些对比给我留下一个越来越深的结论：同样的思路下，体积更大的未必更好。某些组合是真的奇妙，恰好的体积配上恰当的梯度精度策略，效果才是最好的。现在市面上小体积的版本已经很多了，我没有一一去测试。说实话，我做的现在也许不是市场上最佳的了。但我的策略依然有效：一部分取决于底模，另一部分取决于梯度精度策略本身。所以这个文件依然有价值，尤其是对小显存的用户。这就是这套策略用在 Qwen3.8 上的结果：KQV-Q4 + IQ3 Floor 混合量化，每个 K/Q/V 锚点块不低于 Q4_K，每个 norm 保持 F32，FFN 走 IQ3_XXS 地板，重要性矩阵决定哪里降档。内置 MTP。16 GB 显卡 + Q4 KV 缓存，13 万+ 上下文照样装得下。

它的不同之处：带完整 15 张量 MTP 头；16 GB 开 13 万+ 上下文。都装在一个 10 GB 的文件里。

基于 Qwen3.8；感谢 Qwen 团队，感谢底模作者 unsloth。 简单来说，本文件基于 unsloth/Qwen3.8-27B-GGUF 底模权重的混合精度量化，带有 MTP 头。

文件

本模型 SHA256：33442d8a1f49b3ee243034a01826747eed774162c7ffd1811552d23d38377699

混合精度：KQV-Q4 + IQ3 Floor

866 张量，3.18 BPW。与我的其他 KQV 构建同一配方：

类型直方图是 Q4_K 49 · Q6_K 1 · IQ3_S 130 · IQ3_XXS 326 · F32 360，逐张量校验通过。IQ3 级张量用重要性矩阵量化，无矩阵条目的 MTP 头张量则走标准量化。

相对普通 10 GB 档的优势

1. 体积小、锚点全保：一个 ~10 GB 的文件，K/Q/V 锚点全部 ≥ Q4_K，norm 全部 F32。
2. MTP 头在里面，而同档不是每个构建都带。这个带完整 15 张量头部，--spec-type draft-mtp 在聊天长度输出上约 2-3×。
3. 16 GB 开 13 万+ 上下文。同体积 f16-KV 构建只有零头。而且这个上下文下还能把视觉模型一起跑起来：--mmproj 放 CPU 上，速度几乎无影响。

MTP

• block_count = 65、nextn_predict_layers = 1，也就是说 MTP 头是 blk.64 完整块（11 个注意力/FFN 张量 + 4 个 nextn 张量）。
• 头部精度：attn_v = Q6_K（全文件最高精度），其余权重 = IQ3_S（地板上一档），norm 全部 F32。
• 草稿头走在地板上一档，attn_v 保持全文件最高精度。接受率就靠它撑着。

用法

llama.cpp b10728 实测：

llama-server.exe `
  -m Qwen3.8-27B-UD-KQVQ4-IQ3Floor-10G-MTP.gguf `
  -c 131072 -ngl 99 -t 8 --kv-unified `
  --jinja --reasoning on --reasoning-format deepseek `
  --spec-type draft-mtp -np 1 --spec-draft-n-max 2 --spec-draft-p-min 0 `
  --cache-type-k q4_0 --cache-type-v q4_0 -fa on `
  --top-k 20 --top-p 0.95 --repeat-penalty 1.1 --min-p 0

• --spec-type draft-mtp：启用内置 MTP 头（聊天长度输出预期 ~2-3×）。
• --cache-type-k q4_0 --cache-type-v q4_0：Q4 KV 缓存。13 万装进 16 GB 靠的就是它。
• 多模态：用 --mmproj 配 Qwen3.8-27B 视觉投影器。放 CPU 上跑，速度几乎无代价，13 万上下文也装得下。
• LM Studio：直接导入 GGUF，服务端设置里启用投机解码（MTP）。

验证

免责声明

本模型按原样（AS IS）提供，不作任何明示或暗示的保证；以 Apache 2.0 许可证条款为准。本构建为个人研究用途。请自行测试后再使用，风险自负。

底模：unsloth/Qwen3.8-27B-GGUF。

底模权重与许可证归 unsloth（Qwen 团队）。MTP 头与 KQV-Q4 + IQ3 Floor 混合精度量化为本构建新增。 Base weights and license by unsloth (the Qwen team), with thanks. MTP head and mixed-precision quantization added by this build.