Qwen3.5-9B-DFlash — GGUF Quantized Draft Model
DFlash Speculative Decoding optimized for Speculative Decoding acceleration with target models.
This repository contains the GGUF quantized versions of z-lab/Qwen3.5-9B-DFlash, a Dense architecture draft model designed exclusively for Speculative Decoding inference acceleration.
The DFlash model serves as a lightweight draft predictor that rapidly generates candidate tokens, which are then verified by a larger target model (e.g., Qwen3.5-9B). This two-stage approach dramatically improves token generation throughput without sacrificing output quality.
Compatibility
Requires a recent version of llama.cpp with native DFlash support.
Tested with:
- llama.cpp b9831 or newer
Model Summary
| Property | Value |
|---|---|
| Base Model | z-lab/Qwen3.5-9B-DFlash |
| Architecture | Dense (all parameters active at inference) |
| Purpose | Speculative Decoding — draft/verify pipeline |
| Format | GGUF |
| License | Apache 2.0 |
| llama.cpp Requirement | b9831+ (native DFlash support) |
Quantization Summary
| Quant | File | Size | VRAM (~) | Rating |
|---|---|---|---|---|
| BF16 | qwen3.5-9b-dflash-bf16.gguf | 2.4 GB | ~2.5 GB | ★★ |
| Q8_0 | qwen3.5-9b-dflash-Q8_0.gguf | 1.3 GB | ~1.4 GB | ★★★ |
| Q6_K | qwen3.5-9b-dflash-Q6_K.gguf | 1.0 GB | ~1.1 GB | ★★★★ |
| Q5_K_M | qwen3.5-9b-dflash-q5_k_m.gguf | 0.9 GB | ~1.0 GB | ★★★★★ |
| Q4_K_M | qwen3.5-9b-dflash-Q4_K_M.gguf | 0.7 GB | ~0.8 GB | ★★★ |
Quick Reference
- 🏆 Recommended (Sweet Spot):
Q5_K_M— Best balance of accuracy and speed for 16GB VRAM setups - 💾 Smallest:
Q4_K_M— Fits comfortably on any GPU; marginal quality loss - 🔬 Highest Fidelity:
BF16/Q8_0— Near-original precision; useful for benchmarking
Quantization Workflow
All quantizations are performed using llama.cpp’s ./llama-quantize tool. The workflow starts from the BF16 master copy and produces four quantized variants:
Step 1: BF16 Master (Reference)
# Ensure you have a clean BF16 source file
cp qwen3.5-9b-dflash-bf16.gguf ./source-bf16.gguf
Step 2: Generate Quantized Versions
./llama-quantize ./source-bf16.gguf \
./output/qwen3.5-9b-dflash-Q4_K_M.gguf Q4_K_M
./llama-quantize ./source-bf16.gguf \
./output/qwen3.5-9b-dflash-q5_k_m.gguf Q5_K_M
./llama-quantize ./source-bf16.gguf \
./output/qwen3.5-9b-dflash-Q6_K.gguf Q6_K
./llama-quantize ./source-bf16.gguf \
./output/qwen3.5-9b-dflash-Q8_0.gguf Q8_0
Platform Notes
| OS | Command Separator |
|---|---|
| Windows (PowerShell) | ; or backtick ` for line continuation |
| macOS / Linux (Bash) | && or \ for chaining |
★ On Windows PowerShell, use semicolons (
;) or the backtick escape character to chain commands. On macOS/Linux, use&&or backslash-newline.
How to Use
Speculative Decoding with llama-server
llama-server \
--model Qwen3.5-9B.gguf \
--spec-draft-model qwen3.5-9b-dflash-q5_k_m.gguf \
--spec-type draft-dflash \
--spec-draft-n-max 3 \
--host 127.0.0.1 \
--port 8080
Via llama-cli (Interactive)
./llama-cli \
-m Qwen3.5-9B.gguf \
--spec-draft-model qwen3.5-9b-dflash-q5_k_m.gguf \
--spec-type draft-dflash \
--spec-draft-n-max 3 \
-p "Write a concise summary of..." \
-n 512
VRAM Optimization Tips
| GPU VRAM | Recommended Quant | Max Target Model Quant |
|---|---|---|
| 8 GB | Q4_K_M (0.7 GB) | Q3_K_M (~4 GB) |
| 12 GB | Q5_K_M (0.9 GB) | Q4_K_M (~6 GB) |
| 16 GB | Q5_K_M (0.9 GB) | Q5_K_M (~8 GB) |
| 24 GB | Q6_K (1.0 GB) | Q6_K (~18 GB) |
Acknowledgments
- z-lab — Original DFlash model architecture and implementation
- Qwen Team — Qwen3.5-9B base model
- llama.cpp / ggml-org — GGUF format, DFlash inference engine, and quantization toolchain
License
This repository contains converted GGUF quantizations of the original DFlash draft model. All original licenses, usage restrictions, and intellectual property remain with the upstream authors. Refer to the original HuggingFace model card for complete licensing information.
Quantized for speculative decoding acceleration. Always pair with a compatible target model.