Qwen3.8-27B Custom Mixed-Tensor GGUF Quantizations

Two custom mixed-tensor quantizations of Qwen/Qwen3.8-27B-FP8, the dense 27B multimodal model with a hybrid GatedDeltaNet + Gated Attention architecture and Multi-Token Prediction (MTP) head.

Built with llama.cpp at commit 1692f9e, using an importance matrix (imatrix) and per-tensor quantization overrides.

Both quantizations are designed around the 16 GB VRAM budget of consumer GPUs such as the AMD Radeon RX 9070 XT.

On 24 GB GPUs(3090, 4090, 7900 XTX), both quants have substantial headroom — full-context runs with Q8_0 KV cache throughout should fit somewhat comfortably.

PPL comparison

The following comparison was measured against other Q4 quantizations:

NOTE: cHunter789 IQ4_KT and mradermacher IQ4_XS PPL test I didn't run. I used the numbers that cHunter789 provided and they might not match real result since I used a different version of llama-perplexity.

Files

Which one should I pick?

IQ4-MIX

Choose IQ4-MIX if you want the best quality and don't need context beyond ~32K.

IQ3-MIX

Choose IQ3-MIX if you need larger context windows (64K–128K), want more VRAM headroom, or need to run other GPU workloads alongside the model.

Performance

Tested on an AMD Radeon RX 9070 XT (gfx1201, 16 GB VRAM) using LM Studio's Vulkan build:

llama.cpp-linux-x86_64-vulkan-avx2-2.29.0

Generation speed (TG)

IQ3-MIX is approximately 14% faster than IQ4-MIX due to its lower average bits-per-weight and reduced memory bandwidth requirements.

Prompt processing (PP)

Context vs. VRAM

Measured with the indicated KV-cache quantization:

Note: The VRAM figures above depend on the backend, runtime configuration, batch size, and other loaded GPU resources. They should be treated as practical measurements rather than hard minimums.

With Q8_0 + Q4_0 KV cache, IQ3-MIX can reach 128K context on a 16 GB GPU, while IQ4-MIX is more constrained by VRAM at larger context sizes.

Quantization Recipes

Both recipes use an importance matrix (imatrix) computed from 10 chunks × 2048 context of wikitext-103.

IQ4-MIX — Q4, 4.13 BPW

IQ3-MIX — Q3, 3.68 BPW

Both recipes follow the "sensitive trio" philosophy inspired by Unsloth Dynamic:

• output.weight
• attention value projections (attn_v)
• FFN down projections (ffn_down)

These tensors are kept at Q5_K or higher, while more tolerant tensors such as embeddings and linear-attention projections are pushed to lower bit-widths.

Usage

Text + Vision

RADV_PERFTEST=nogttspill llama-server \
  -m Qwen3.8-27B-IQ4-MIX.gguf \
  --mmproj mmproj-F16.gguf \
  -ngl 999 -c 16384 -b 2048 -ub 2048 -t 12 -np 1 \
  --cont-batching --jinja --flash-attn on \
  --cache-type-k q8_0 --cache-type-v q8_0 --kv-unified \
  --host 0.0.0.0 --port 8080

Replace IQ4-MIX with IQ3-MIX when using the Q3 version.

For larger context windows, increase -c. IQ3-MIX supports up to 128K context with Q8_0 + Q4_0 KV cache under the tested configuration.

Download

hf download vmarcelo/Qwen3.8-27B_IQ4-MIX_GGUF \
  --include \
    "Qwen3.8-27B-IQ4-MIX.gguf" \
    "Qwen3.8-27B-IQ3-MIX.gguf" \
    "mmproj-F16.gguf" \
  --local-dir ./

About the Source Model

Qwen3.8-27B-FP8 is a dense multimodal model with the following architecture:

• 64 transformer layers + 1 MTP layer

• Hybrid attention:

  • 48 layers use linear attention (GatedDeltaNet-style SSM)
  • 16 layers use standard full attention

• Vision encoder

• Multi-Token Prediction (MTP) head

• Vocabulary size: 248,320

• Hidden size: 5,120

• Context length: 262,144 tokens

Every fourth transformer layer uses standard full attention, with the remaining transformer layers using the linear-attention architecture.

All supported model features, including MTP and multimodal vision, are preserved in both GGUFs.

Acknowledgements

• Source model: Qwen/Qwen3.8-27B-FP8 — Qwen team
• Quantization tool: llama.cpp — ggml-org
• Vulkan build: LM Studio
• mmproj: unsloth/Qwen3.8-27B-GGUF