Qwen3.8-27B GGUF
Source-faithful and architecture-aware GGUF conversions of
Qwen/Qwen3.8-27B, including its
native vision projector and one-layer MTP draft model.
This release was independently built from source revision
1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
with llama.cpp revision
1692f9e50bb20fd96b963af38a282daf78feea64.
All 18 source safetensors shards were verified against their Hub LFS SHA-256
digests before conversion.
Files
Main model
| File | Size | Purpose |
|---|---|---|
Qwen3.8-27B-Q4_K_M.gguf | 18.97 GB | Recommended practical model. Conservative 5.64 BPW mixed quantization. |
Qwen3.8-27B-Q8_0.gguf | 28.60 GB | Near-lossless high-quality quantization. |
Qwen3.8-27B-BF16.gguf | 53.81 GB | Maximum-fidelity BF16 conversion. |
The recommended Q4 file is deliberately not a stock Q4_K_M quant. Its FFN and embedding matrices use Q4_K, the final output and attention-output projections use Q6_K, and all other attention plus Gated DeltaNet/SSM matrices use Q8_0. This protects the model's architecture-sensitive paths while keeping the file under 19 GB.
Vision projector
Download one projector to use images or video. It is not needed for text-only inference.
| File | Size | Purpose |
|---|---|---|
mmproj-Qwen3.8-27B-Q8_0.gguf | 0.63 GB | Recommended practical projector. |
mmproj-Qwen3.8-27B-BF16.gguf | 0.93 GB | Maximum-fidelity projector. |
MTP speculative-decoding sidecar
These files are optional. They enable Qwen3.8's native one-layer multi-token-prediction draft model; they do not change the target model's final sampling distribution.
| File | Size | Purpose |
|---|---|---|
mtp-Qwen3.8-27B-Q4_0.gguf | 1.68 GB | Smallest and usually best practical draft sidecar. |
mtp-Qwen3.8-27B-Q8_0.gguf | 3.16 GB | Higher-fidelity draft sidecar. |
mtp-Qwen3.8-27B-BF16.gguf | 5.95 GB | Maximum-fidelity draft sidecar. |
Download
Recommended text + vision pair:
hf download Mike0021/Qwen3.8-27B-GGUF \
Qwen3.8-27B-Q4_K_M.gguf \
mmproj-Qwen3.8-27B-Q8_0.gguf \
--local-dir ./Qwen3.8-27B-GGUF
Add mtp-Qwen3.8-27B-Q4_0.gguf to that command if you want speculative
decoding.
llama.cpp usage
This new architecture requires llama.cpp revision 1692f9e or a tested newer
revision. Other GGUF runtimes and older GUIs may not support Qwen3.8 yet.
Text chat with thinking disabled:
llama-cli \
-m Qwen3.8-27B-Q4_K_M.gguf \
-ngl all -c 32768 --jinja --conversation \
--chat-template-kwargs '{"enable_thinking":false}'
Image understanding:
llama-cli \
-m Qwen3.8-27B-Q4_K_M.gguf \
--mmproj mmproj-Qwen3.8-27B-Q8_0.gguf \
--image image.jpg \
--image-min-tokens 1024 \
-p "Describe this image precisely." \
-ngl all -c 32768 -n 512 --jinja
MTP speculative decoding:
llama-cli \
-m Qwen3.8-27B-Q4_K_M.gguf \
-md mtp-Qwen3.8-27B-Q4_0.gguf \
--spec-type draft-mtp --spec-draft-n-max 3 \
-ngl all -ngld all -c 32768 --jinja --conversation \
--chat-template-kwargs '{"enable_thinking":false}'
The model's native context window is 262,144 tokens. Start with a smaller runtime context such as 32K unless you need the full window, because KV and recurrent-state memory grow with the configured context and concurrency.
Thinking and sampling
The exact source Jinja template is embedded in every text and MTP GGUF. Thinking
defaults to xhigh. Select another supported effort explicitly:
--chat-template-kwargs \
'{"enable_thinking":true,"reasoning_effort":"low","preserve_thinking":true}'
Supported effort values are low, medium, and xhigh. Use
{"enable_thinking":false} to disable thinking. At the pinned llama.cpp
revision, pass the effort through chat_template_kwargs; top-level OpenAI API
reasoning_effort forwarding is still being completed in
llama.cpp PR #26941.
Qwen's source model card recommends these starting points:
| Mode | Temperature | Top-p | Top-k | Presence penalty |
|---|---|---|---|---|
| Thinking | 1.0 | 0.95 | 20 | 0.0 |
| Non-thinking | 0.7 | 0.8 | 20 | 1.5 |
Disabling thinking does not automatically change the sampler settings.
Validation and provenance
The release passed the following checks before upload:
- GGUF v3 metadata:
qwen35, 64 target layers, 262,144 context, 248,320-token vocabulary. - Exact source chat-template comparison.
- Expected tensor counts: 851 text, 334 projector, and 18 MTP tensors.
- Expected type distributions for every BF16, Q8, and Q4 artifact.
- Full SHA-256 checksums (see
SHA256SUMS). - CUDA tensor checks and deterministic text-generation smoke tests.
- Native image/projector and MTP speculative-decoding smoke tests.
The exact conversion commands, tool revisions, tensor policy, and validation
details are in CONVERSION.md. Raw conversion and
quantization logs are included under logs/.
Why no importance-matrix Q4?
llama.cpp supports importance-matrix quantization, but the architecture was new at conversion time and no representative Qwen3.8-specific calibration plus held-out evaluation was available. An unvalidated calibration corpus could bias the remaining Q4 FFNs. This release therefore follows ggml-org's conservative, architecture-aware reference recipe, which already keeps the attention and DeltaNet/SSM paths at Q8. Any future imatrix build should be published as a separate variant with its calibration provenance and held-out KLD/task results.
License
The original model and these converted weights are distributed under the
Apache License 2.0. See the
Qwen/Qwen3.8-27B model card for
the upstream model's documentation and intended-use guidance.