Qwen3.6-27B-MTP TQ4_1S GGUF
This repository contains a TurboQuant/TQ4_1S GGUF quantization of Qwen/Qwen3.6-27B.
[!WARNING] This GGUF will not work with original upstream
llama.cpp. It requires a TurboQuant-enabled build with TQ4_1S support, such asTheTom/llama-cpp-turboquant.
The source GGUF used for this quantization was Qwen3.6-27B-MTP-BF16.gguf. The final artifact keeps the MTP/next-token-prediction block in BF16 because quantizing that block was found to break MTP behavior.
File
| File | Size | SHA256 |
|---|---|---|
Qwen3.6-27B-MTP-TQ4_1S.gguf | 21,403,947,936 bytes | 1e9303f1e00daf4a85ad04fe9f31738526fb00bbafa944b0b68f2fba6aa4440e |
Quantization Policy
This model was quantized with a llama.cpp TurboQuant fork that supports TQ4_1S weight tensors.
Workflow:
- BF16 source GGUF to mixed Q8_0 source GGUF.
blk.64/nextn/ MTP tensors preserved as BF16 during the Q8_0 pass.- Final tensor-role-aware hybrid TQ4_1S pass from the mixed Q8_0 source.
Final tensor mix:
| Tensor type | Count | Notes |
|---|---|---|
| TQ4_1S | 180 | Middle-layer full-attention Q/K/V/O and FFN gate/up tensors |
| Q4_K | 60 | Middle-layer FFN down tensors |
| Q8_0 | 258 | Embeddings, output, SSM/Mamba tensors, protected boundary tensors |
| BF16 | 8 | blk.64 MTP-critical tensors |
| F32 | 360 | Norms and small scalar tensors |
Protected layout:
- Main text blocks:
blk.0throughblk.63 - MTP/nextn block:
blk.64 - Protected boundary blocks:
blk.0,blk.1,blk.62,blk.63 - Quantized middle blocks:
blk.2throughblk.61 blk.64is preserved for MTP correctness
The BF16-preserved blk.64 tensors are:
blk.64.attn_k.weight
blk.64.attn_output.weight
blk.64.attn_q.weight
blk.64.attn_v.weight
blk.64.ffn_down.weight
blk.64.ffn_gate.weight
blk.64.ffn_up.weight
blk.64.nextn.eh_proj.weight
The nextn norm tensors remain F32.
Compatibility
This GGUF uses TQ4_1S tensor type support from llama-cpp-turboquant. Standard upstream llama.cpp builds that do not know the TQ4_1S GGUF tensor type may fail to load this file.
Use a build with TQ4_1S weight support, such as the TurboQuant llama.cpp fork.
Example:
./llama-server \
-m Qwen3.6-27B-MTP-TQ4_1S.gguf \
-ngl 99 \
-c 131072 \
--cache-type-k q8_0 \
--cache-type-v turbo3
For a more conservative KV cache setting, use --cache-type-k q8_0 --cache-type-v q8_0.
Provenance
- Parent model: Qwen/Qwen3.6-27B
- Source format: BF16 GGUF with MTP tensors
- Quantizer: llama.cpp TurboQuant fork with TQ4_1S support
- Final GGUF:
Qwen3.6-27B-MTP-TQ4_1S.gguf
Notes
This is an experimental GGUF quantization. Validate quality and MTP behavior on your workload before production use.