Qwen3.8-27B GGUF

GGUF quantizations of Qwen/Qwen3.8-27B, made with llama.cpp.

Chinese version: README_zh.md

A 27B dense vision-language model (27.8B including the vision tower). 64 decoder layers in a hybrid stack: full_attention_interval=4 means every fourth layer is grouped-query full attention (16 layers) and the rest are Gated DeltaNet linear attention (48 layers). Vocab 248320 with untied embeddings. Context length 262144 natively, extensible further with YaRN.

Text, image and video in; text out.

What is in this repo

The mmproj file is separate by design: llama.cpp loads the text backbone and the vision projector as two files. Download one tier plus one mmproj. Without the mmproj the model still runs, but text-only.

The mmproj- and mtp- prefixes are what recent llama.cpp builds match on to tell the three file kinds apart, so -hf qtum/Qwen3.8-27B-GGUF picks a backbone rather than a projector, and --spec-type mtp finds the draft on its own.

The draft is larger than an MTP head alone would suggest: a standalone draft needs its own embedding and output projection, and with vocab 248320 × hidden 5120 untied those two copies dominate the file. The MTP block itself is small.

Quantizations

Tiers a smaller file already matches

The ladder is published in full for coverage, which means a few tiers end up larger than another tier that measured no worse on this model:

This is a size argument, not a quality one. The size gap is real; the PPL side is a tie — Q5_K_M and Q5_K_S differ by 0.004 against a ±0.074 error bar, and Q4_1 vs Q4_K_M by 0.0002. Nothing here says the left column is broken.

Q4_0 and Q4_1 are the pre-K-quant formats and IQ4_NL targets backends without full K-quant support, so they are kept for runtimes that need them. If yours loads either, take the right-hand column.

The master row is not a file in this repo. It is listed so the numbers above have a reference point.

KV cache is cheaper than the layer count suggests

Only the 16 full-attention layers keep a growing KV cache; the 48 linear-attention layers hold a fixed-size recurrent state that does not grow with context. That works out to roughly 64 KiB per token, so 32k context costs about 2 GiB — much less than a 64-layer all-attention model of this size would need.

Budget roughly tier size + 0.9 GiB (mmproj) + KV cache when picking a tier.

Calibration and protection

Every tier is imatrix-calibrated (n_ctx=512, mixed code / English / Chinese), and imatrix.gguf is published here so the calibration is auditable and reusable. The layers that suffer most under low-bit compression are protected:

Deliberately not pinned: ssm_out, attn_qkv and attn_gate take the tier type like any other projection. Together they are 20.6% of the backbone, so an 8-bit floor under all three flattens the ladder: a Q4_K_M built that way came out at 22.97 GiB / 7.33 bpw against the shipped 16.53 GiB / 5.28 bpw, with five tiers landing within 7 GiB of each other.

Both builds were measured on the same command. The heavy one scored 6.8161 against 6.8103 shipped — a tie inside the ±0.074 error bar. The extra 6.44 GiB bought nothing. ssm_out reads like state but is the output projection of the linear-attention block, the counterpart of attn_output; llama.cpp's own quantizer gives it no special treatment either.

token_embd and output are 9.4% of the backbone on their own — vocab 248320 × hidden 5120, and the two are untied so there are two copies. Pinning both to Q6_K at every tier would push the low tiers well above their nominal bit-width (IQ2_XS would land near 3.0 bpw effective), so at sub-4-bit tiers they drop to Q4_K instead. The BPW column above is the effective figure, protection included.

Usage

# Text only
llama-cli -m Qwen3.8-27B-Q4_K_M.gguf -c 8192

# With vision — pass the projector alongside the backbone
llama-mtmd-cli -m Qwen3.8-27B-Q4_K_M.gguf \
  --mmproj mmproj-Qwen3.8-27B-Q8_0.gguf \
  --image photo.jpg -p "Describe this image."

Do not pass -ngl manually. llama.cpp fits layers to free VRAM by itself; setting -ngl explicitly aborts that fitting (n_gpu_layers already set by user) and can silently fall back to CPU.

Speculative decoding with the MTP head. --spec-type draft-mtp is required, not optional: with -md alone the draft is loaded but never used, because the speculative type stays unset and the MTP path is gated on it.

llama-server -m Qwen3.8-27B-Q4_K_M.gguf \
  -md mtp-Qwen3.8-27B-Q8_0.gguf --spec-type draft-mtp

Pulling straight from this repo, the draft is discovered from the mtp- prefix and the type is inferred, so --spec-type can be omitted:

llama-server -hf qtum/Qwen3.8-27B-GGUF --spec-draft-hf qtum/Qwen3.8-27B-GGUF

About the PPL numbers

Measured in-house with one fixed setup, master baseline run through the exact same command. These numbers are only comparable within this table. Do not compare them against PPL figures published by other repos — different corpora and chunk counts make absolute values meaningless across setups.

License

Apache 2.0, inherited from Qwen/Qwen3.8-27B. Quantized by qtum.