How to Run Qwen3.8 27B Locally

Every quantization of this model we could find, ours and everyone else's, measured against the original weights on the same held-out text. The dashed line is the best file available at each size from anyone but us.

Pick a file

Every number below is measured, not estimated. How we measured it is at the bottom, and the raw logs are in the metrics repo so you can check any of it yourself.

KL divergence is how far the quantized model's predictions drift from the original weights. Lower is better, and zero means identical. top-1 is how often it picks the same next word the original would have picked.

AD- marks an Atomic Dynamic layout. The name says what the two largest tensor groups got: AD-<ffn_down>-<ffn_up>, collapsed to one name when both match. Nothing is named after a type it does not contain.

Which one fits your card

The file has to fit, and so does the context. This model keeps 256 KB of attention cache per token, which is 2 GB at 8k context and 8 GB at 32k. Budget for both.

[!TIP] If you are choosing between two neighbouring files, take the larger one. The steps between them cost one or two gigabytes and buy noticeably fewer wrong words, especially below 14 GB where the curve gets steep.

Running it

llama-server -m Qwen3.8-27B-AD-Q4_K_M.gguf -ngl 99 -c 8192

Prompt format:

<|im_start|>system
{system}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
<think>

The model ships a multi token prediction head. It is inside every file here and needs no extra download:

llama-cli -m Qwen3.8-27B-AD-Q4_K_M.gguf --spec-type draft-mtp -ngl 99 -c 8192

Speed

Measured on 2xRTX_5090 with AD-Q4_K_M, full offload:

Images and video

This model reads images, not only text. That needs one extra file, the vision projector, which is separate from the quant and shared by all of them:

• mmproj-Qwen3.8-27B-F16.gguf for most people
• mmproj-Qwen3.8-27B-BF16.gguf if your setup prefers bfloat16

Download it once alongside whichever quant you picked:

llama-mtmd-cli \
  -m Qwen3.8-27B-AD-IQ4_XS.gguf \
  --mmproj mmproj-Qwen3.8-27B-F16.gguf \
  --image your-photo.jpg --image-min-tokens 1024 \
  -ngl 99 -c 8192 \
  -p "What is in this image?"

--image-min-tokens 1024 is not optional in practice. llama.cpp warns that this family needs at least that many image tokens or anything positional gets unreliable.

How well it survives quantization

demo.jpg in this repo is a piece of handwritten calligraphy in a gothic hand. It is a harder test than a photograph: a misread word is obvious, so you can score the answer yourself instead of taking our word for it.

The text on it reads:

Peace of mind comes to me through making things with my hands. An added bonus comes if my efforts inspire others to try the creative process.

Every file from AD-IQ3_S upward transcribed that exactly, line breaks included. Vision holds up better under quantization than the language half does, which surprised us. Full transcripts for every file are in the metrics repo under logs/vision-*.log.

Try it yourself on the same image:

llama-mtmd-cli -m <your-quant>.gguf --mmproj mmproj-Qwen3.8-27B-F16.gguf \
  --image demo.jpg --image-min-tokens 1024 -ngl 99 -c 8192 \
  -p "Transcribe the text in this image exactly, then describe the decoration."

[!NOTE] While an image is being encoded llama.cpp prints lines about find_slot: non-consecutive token position and about unused blk.64 tensors. Both are expected. The first is how this family numbers image patches, using a multi dimensional rope rather than one running index. The second is the multi token prediction head, which a plain forward pass does not run.

How these compare to other builds

Numbers taken against different references cannot be put in the same table, so we did not copy anyone's published figures. We downloaded their files and measured them ourselves, against the same original weights, on the same held-out text. Their logs are in the metrics repo next to ours.

At sizes where the comparison is direct:

[!NOTE] One honest exception. At 17.9 GB unsloth's UD-Q4_K_XL reaches 0.00955, which is better than our 17.1 GB file and close to our 18.6 GB one. Around 18 GB their build and ours are within a few percent of each other. We are ahead across most of the range, not all of it.

One thing worth seeing on its own. Three publishers ship a file called Q4_K_M, and they are not the same file:

Same name, two gigabytes apart, and nearly a factor of two in accuracy. A quant name tells you which recipe was requested, not what you are getting.

What we found while building these

Where the bits go matters more than how many there are

We built ten versions of the same 17 to 19 GB file, changing only which tensors got the extra bits, and measured each one against the original weights.

Half the divergence disappears at the same file size, purely from moving bits around. Every file in the ladder above uses the last layout.

The ends of the network are worth more than the middle

Lifting the first four and last twelve layers helped more than anything else we tried. Widening that band to 32 layers did not help further, and spending the same bits on the output head helped less. The importance matrix agrees: the highest activation energy in the whole model sits on layers 52 to 62, with a second peak on layer 0.

This model is a hybrid, and two small groups carry a lot

Alongside ordinary attention, Qwen3.8 has an attention gate and a state output path. They are 5.5% of the weights each. Giving both one extra step of precision cost 0.16 GB and removed 11% of the remaining divergence. That was the single best trade we found.

The embedding table is cheaper than it looks

The token embedding table and the output head weigh the same, 4.7% each, and behave nothing alike. The head decides the next word directly and has to stay precise. The embedding table is a lookup whose error stays inside one token, so it can be cut hard. Paying for the head out of the embedding table is a net gain at every size we tested.

Q8_0 is not lossless

It is very close, but it is not the original: 0.00064 divergence and 98.92% top-1 agreement. We say so because our whole table is measured against the real BF16 weights rather than against Q8_0, and that changes every number in it. Anyone measuring against a Q8_0 reference will get smaller numbers than these for the same files.

The prediction head collects no calibration data

The multi token prediction head is never executed during a normal forward pass, so the importance matrix has nothing to say about it at any corpus size. Quantize it low and llama.cpp refuses partway through rather than guess. It is pinned to q5_k in every file here.

Every tensor row divides by 256

Worth checking before you plan a ladder. K and I quants store weights in superblocks of 256, and a model whose rows do not divide by that number silently falls back to a coarser type while keeping the name you asked for. On this model everything divides cleanly, so the whole range from IQ1_M upward is genuinely available. That is not true of every recent release.

The calibration data

Our importance matrix comes from a corpus we built for this model and published: AtomicChat/calib-corpora, recipe qwen3.8-27b. It is 4,967,044 tokens across 3,004 documents:

Two details that matter more than the mix.

Every conversation is rendered through this model's own chat template, so <|im_start|>, <think>, <tool_call> and <tool_response> appear as the single tokens the model actually sees. Text rendered for another model is excluded from the build rather than reused.

The vocabulary sweep is regenerated for this tokenizer. It covers 246,919 of the 247,133 vocabulary entries that can stand alone, at about two tokens per entry. A sweep built for a different model covers a different vocabulary and does nothing here.

[!IMPORTANT] If you build your own importance matrix from this corpus, pass --parse-special to llama-imatrix. Without it the chat markup is read as ordinary punctuation, and the agentic and reasoning parts of the corpus calibrate on text the model never sees.

How we measured

Reference: the original BF16 weights, converted to GGUF and run unquantized. Perplexity on the held-out set is 4.5219 plus or minus 0.0238.

Held-out text: eval_neutral from our calibration dataset, never used for calibration. 87 chunks at 4096 context.

Metric: per-token KL divergence of each file's predictions against the reference, plus top-1 agreement.

Hardware: 4x RTX 5090, CUDA 13.0. Two cards would have been enough: the BF16 file is 51 GB and the logit buffer adds about 2 GB. I used only 2/4 cards for all the measurements though, just the servers had 4, so i write about it openly to avoid any confusion.

The raw logits of the reference are published in the metrics repo, split into parts because of the file size limit. With them you can measure your own build against exactly the same point we did:

cat base-neutral.kld.*.part > base-neutral.kld

llama-perplexity -m your-quant.gguf -f eval_neutral.txt \
  --kl-divergence-base base-neutral.kld --kl-divergence -c 4096 -ngl 99

Reproducing a file

The importance matrix was collected on the BF16 weights rather than on a quantized copy, split across seven workers and merged. Splitting by chunk range gives the same result as one long run, because the statistic is a sum:

llama-imatrix -m Qwen3.8-27B-bf16.gguf -f calib_train.txt -o shard-0.gguf \
  -ngl 99 -c 512 -b 4096 -ub 4096 --parse-special --from-chunk 0 --chunks 1400

llama-imatrix -m Qwen3.8-27B-bf16.gguf --in-file shard-0.gguf,shard-1.gguf,... \
  -o imatrix.gguf

Then, for AD-Q4_K_M:

llama-quantize --imatrix imatrix.gguf \
  --tensor-type 'blk\.64\.=q5_k' \
  --tensor-type 'blk\.([0-3]|5[2-9]|6[0-3])\.ffn_.*=q5_k' \
  --tensor-type 'blk\.([4-9]|1[01])\.ffn_.*=q5_k' \
  --tensor-type ffn_down=q4_k --tensor-type ffn_gate=q4_k --tensor-type ffn_up=q4_k \
  --tensor-type attn_q=q4_k --tensor-type attn_gate=q5_k --tensor-type ssm_out=q5_k \
  --tensor-type output=q6_k --tensor-type token_embd=iq4_xs \
  Qwen3.8-27B-bf16.gguf Qwen3.8-27B-AD-Q4_K_M.gguf Q8_0

The rules for every other file are in its quantization log in the metrics repo.

Also in this repo

demo.jpg is the calligraphy used in the vision section, so anyone can run the same test on the same image. Its source and licence are in demo-source.txt.

The stock Q4_K_M, Q5_K_M and Q6_K mixes were built from the same importance matrix and measured against the same reference, as a control for the layout comparison above. Their measurements are in the metrics repo. The files themselves are not kept here: at every size the AD- build of the same weight is better, so there is no reason to download one.

Model details

64 layers, hidden size 5120, feed-forward size 17408, vocabulary 248,320, context 262,144, no sliding window. Hybrid attention with an added gate and a state path. One multi token prediction head.

Needs a recent llama.cpp with qwen35 architecture support. Built and tested with llama.cpp b10454.