Qwen3.8-27B pure IQ4_XS: +12% speed, a measured quality step below UD
The speed pick for lucebox serving of Qwen3.8-27B on bandwidth-bound GPUs (measured on one AMD Radeon AI PRO R9700). It is bartowski's imatrix IQ4_XS requantized with llama-quantize --allow-requantize --pure, which drops the mixed-precision quality hedge so every tensor is IQ4_XS: 13.54 GiB instead of 14.50, and decode reads fewer bytes per token through the fastest kernel path on RDNA4.
Know the trade before you download. Measured against a Q8_0 reference (300x512-chunk wikitext-2 paired KL divergence; HumanEval-164 and GSM8K-200 through the serving path, exact-verify speculation so the drafter cannot change outputs):
| this file (pure) | Unsloth UD-IQ4_XS (recommended default) | |
|---|---|---|
| Size | 13.54 GiB | 13.27 GiB |
| HumanEval decode, DFlash2 block 16 | 235.8 tok/s (257.3 peak) | 208.1 tok/s |
| End-to-end, same ten prompts | 174.5 tok/s | 156.2 tok/s |
| Plain decode | 37.4 tok/s | 32.3 tok/s |
| PPL vs Q8_0 | +1.9% | +0.7% |
| Mean KLD vs Q8_0 | 0.028 | 0.018 |
| Top-1 agreement with Q8_0 | 92.0% | 94.1% |
| HumanEval pass@1 | 144/164 | 151/164 |
| GSM8K (first 200) | 171/200 | 177/200 |
Pro: about 12% more tokens per second on every request than the UD file, and the fastest Qwen3.8-27B target we have measured on this class of hardware.
Con: the quality cost is small but real and shows up on every instrument: KL divergence rises by half, top-1 agreement gives up two points, and it hands back seven HumanEval and six GSM8K problems against the UD file. If your workload is quality-sensitive, serve UD-IQ4_XS instead; it is what the lucebox quickstart recommends.
Full methodology, exactness verification, and the llama.cpp comparison: the blog post.
Serve it
./dflash_server Qwen3.8-27B-IQ4_XS-pure.gguf \
--draft qwen38-dflash2-q8_0.gguf \
--draft-block-size 16 --max-ctx 131072 \
--cache-type-k q8_0 --cache-type-v q8_0
The drafter is z-lab's DFlash2 converted with server/scripts/convert_dflash_to_gguf.py from the lucebox repo. Reproduce this file yourself from bartowski's IQ4_XS with:
llama-quantize --allow-requantize --pure \
Qwen3.8-27B-IQ4_XS.gguf Qwen3.8-27B-IQ4_XS-pure.gguf IQ4_XS
Vision: images in, with Qwen3.8-27B-mmproj-Q8_0.gguf
This repo also carries the image encoder (projector) for lucebox image input. It is unsloth's BF16 projector with its weight matrices recast to Q8_0 (rows that are not a multiple of 32 stay F16): 629 MB instead of 931 MB, and it encodes a 975-token image in 443 ms instead of 677 ms on the R9700, with the same scores.
luce_server Qwen3.8-27B-IQ4_XS-pure.gguf --target-device hip:0 \
--draft qwen38-dflash2-q8_0.gguf --draft-device hip:0 \
--mmproj Qwen3.8-27B-mmproj-Q8_0.gguf
Images go in OpenAI-style image_url parts of a chat message, up to four per request. Each image costs one token per 32x32 pixels, 64 to 1,024 tokens. Needs a lucebox build with image input (PR #722); the full guide is docs/image-input.md in the repo.
Measured on one R9700 with this file, the Q8_0 projector and the DFlash2 drafter, thinking off:
| this file + Q8_0 projector | UD-IQ4_XS + BF16 projector | |
|---|---|---|
| AI2D (100 seeded questions) | 90 | 85 |
| ChartQA augmented / human (60 + 60) | 56 / 42 | 55 / 43 |
| Image prompt prefill, average | 0.56 s | 0.71 s |
| One to four images per request | all correct | all correct |
| Image request decode | 36 tok/s | 31 to 35 tok/s |
Text requests are byte-identical with the projector loaded, at the same speed. Image requests decode without the drafter.
sha256 47b8f0575185fcf09a6344fa46288f3506b4134cab77fa09ebb5a88c38d665e5
Credit: unsloth for the projector; bartowski for the imatrix calibration this file inherits; Qwen for the model.