Qwen3.8-9B — GGUF
Developed by Empero
GGUF quantizations of empero-ai/Qwen3.8-9B — a full-parameter distillation of Qwen3.8 2.4T A95B into the Qwen3.5-9B architecture — for llama.cpp, Ollama, LM Studio, Jan, KoboldCpp, and other stock GGUF runtimes.
This card is about choosing a file and running it. The capability writeup, full benchmark results, and best practices live on the main model card.
Headline results for the source model (CoT protocols, lm-evaluation-harness, identical settings base vs. student):
| Task | Qwen3.5-9B (base) | Qwen3.8-9B | Δ |
|---|---|---|---|
| mmlu (CoT, 57 subjects) | 0.546 | 0.751 | +0.205 |
| gsm8k_cot | 0.885 | 0.870 | −0.015 |
[!Note] Qwen3.5-class models are hybrids: three Gated DeltaNet layers for every full-attention layer. A recent llama.cpp build with Qwen3.5 / Gated DeltaNet support is required — older builds will fail to load the architecture.
Files
| File | Quant | Size | Notes |
|---|---|---|---|
Qwen3.8-9B-Q4_K_M.gguf | Q4_K_M | 5.780 GB | Recommended. Best quality/size balance for most users. |
Qwen3.8-9B-Q5_K_M.gguf | Q5_K_M | 6.643 GB | Higher quality, still fits an 8 GB card at short context. |
Qwen3.8-9B-Q6_K.gguf | Q6_K | 7.559 GB | Near-lossless. |
Qwen3.8-9B-Q8_0.gguf | Q8_0 | 9.786 GB | Highest-quality quantization. |
Qwen3.8-9B-BF16.gguf | BF16 | 18.407 GB | Full precision reference. |
Sizes are exact decimal GB from the uploaded files (1 GB = 1,000,000,000 bytes).
What fits on a GPU?
Practical weight-size-based guidance at modest context — the KV cache is the dominant cost at long context and may require offload regardless of weight quant:
| Quant | Guidance |
|---|---|
| Q4_K_M / Q5_K_M | Comfortable on 8–12 GB cards for everyday use. |
| Q6_K / Q8_0 | 12–16 GB recommended. |
| BF16 | 24 GB+. |
Usage
llama.cpp
llama-cli -m Qwen3.8-9B-Q4_K_M.gguf \
--temp 0.6 --top-p 0.95 --top-k 20 \
-n 16384 -cnv
Use the built-in chat template (-cnv). The model is a reasoning model: every answer opens with a <think> block, so allow a generous -n and strip the <think>...</think> span for end users.
Ollama / LM Studio / Jan / KoboldCpp
Download the GGUF of your choice and load it directly; the chat template is embedded in the file. Recommended sampling: temperature=0.6, top_p=0.95, top_k=20.
Provenance & licensing
Quantizations of empero-ai/Qwen3.8-9B, a distillation of Qwen3.8 2.4T A95B into Qwen/Qwen3.5-9B trained on ~70,000 curated teacher traces from our internal Qwen3.8 distillation datasets. Weights are Apache-2.0, inherited from the Qwen base, shared as-is.
Stay in the loop
Sign up for the Empero newsletter at empero.org for releases, evals, and research notes.
Support / Donate
If this model helped you, consider supporting the project:
- BTC:
bc1qx6zepu6sfkvshgdmc4ewu6pk6rpadvpgffpp7v - LTC:
ltc1qv2mefzps2vtjcpwfx8xxdrpplrcvltswm68r7x
Acknowledgements
- Developed and released by Empero
- Base model: Qwen3.5-9B (Alibaba Qwen team)
- GGUF quantization: llama.cpp (ggml-org)