Qwen3.8-27B-ED-5bpw-GGUF
Dear visitors,
I would like to present to you the Qwen3.8-27B model, quantized to a size of 5 BPW. No ablation or post-training was performed on this model. The ED-5bpw model already features an embedded MTP layer, so there is no need to download it separately. This version differs from the original only by its quantization using an Importance Matrix. During compression, the model was calibrated on the following topics:
- Russian technical language.
- Programming languages:
c, c++, objective-c, bash, python, javascript, typescript. Formal logicand constructing logical connections.
The source model used was the BF16 quantization file from ggml-org. The model is designed to run on a modern consumer graphics card with 16 GB of VRAM, such as the RX 9070 XT. On this hardware, inference speeds average around 20-30 t/s, depending on the KV cache fill level.
Statistics
Below are the accuracy measurements within the general weight category compared to Qwen3.8-27B-GGUF:UD-Q4_K_XL by Unsloth AI. The statistics were obtained using the llama-perplexity tool. The measurements were conducted using the exact same Importance Matrix that was applied during the creation of the quantized model version.
| Statistics | UD-Q4_K_XL (16.34 GiB) | ED-5bpw (15.92 GiB) | Q8_0 (26.62 GiB) |
|---|---|---|---|
| Cor(ln(PPL(Q)), ln(PPL(base))) | 97.78% | 98.12% | 99.10% |
| Mean PPL(Q)-PPL(base) | 0.010224 | 0.009144 | 0.013471 |
| Mean KLD | 0.028143 | 0.027261 | 0.007148 |
| Maximum KLD | 31.597418 | 31.554943 | 24.629694 |
| 99.9% KLD | 6.144548 | 6.126769 | 0.773549 |
| Mean Δp | -0.120 | -0.128 | -0.031 |
| RMS Δp | 4.530 | 4.363 | 1.802 |
| Same top p | 97.574 | 97.573 | 99.215 |
Benchmark
Also included below are the comparative benchmark results obtained using the llama-bench utility.
| test | UD-Q4_K_XL (16.34 GiB) t/s | ED-5bpw (15.92 GiB) t/s | Q8_0 (26.62 GiB) t/s |
|---|---|---|---|
| pp512 | 524.39 ± 0.00 | 1238.60 ± 0.00 | 319.07 ± 0.00 |
| tg128 | 24.16 ± 0.00 | 25.29 ± 0.00 | 4.35 ± 0.00 |
| pp512 @ d16384 | 470.08 ± 0.00 | 1080.62 ± 0.00 | 297.60 ± 0.00 |
| tg128 @ d16384 | 16.90 ± 0.00 | 20.01 ± 0.00 | 4.03 ± 0.00 |
| pp512 @ d32768 | 430.21 ± 0.00 | 946.29 ± 0.00 | 279.84 ± 0.00 |
| tg128 @ d32768 | 14.30 ± 0.00 | 16.48 ± 0.00 | 3.86 ± 0.00 |
Additional Information
Working with a context size greater than 90k is practically useless, as the inference speed drops to 15 t/s.