Qwen3.8-27B-TQ
Pending engine release - do not use yet. These files need the TQLLM engine, which is not released yet. This page will say when it is.
Qwen3.8-27B for the TQLLM engine, sized for 16 GB GPUs, with a DFlash2 draft model for exact speculative decoding.
Credits
| What is in these files | Credit |
|---|---|
| Qwen3.8-27B weights, MTP head, chat template | Qwen team, Alibaba (Qwen) |
| DFlash2 draft model weights and method | Jian Chen, Yesheng Liang, Zhijian Liu, z-lab (arXiv 2602.06036) |
| BF16 source GGUF | unsloth/Qwen3.8-27B-GGUF |
| Fixed chat template (embedded) | froggeric, Qwen-Fixed-Chat-Templates |
| Rotation-based quantisation, the lineage of the TQ3 / TQ4 formats | Google Research, TurboQuant |
Credits for the engine (KV cache, speculative decoding, kernels) will be in the TQLLM README when the engine source is released.
Files
| File | Size | What it is |
|---|---|---|
Qwen3.8-27B-TQ_S.gguf | 12.55 GB | the 27B model (TQ3 / TQ4 tensors) |
drafters/Qwen3.8-27B-DFlash2-TQ3.gguf | 0.79 GB | DFlash2 draft model, TQ3 weights |
mtp-vocab/atx_65536.txt | 0.4 MB | the draft model's token list (65,536 ids) |
drafters/NOTICE | licence notice for the draft model |
hf download YTan2000/Qwen3.8-27B-TQ Qwen3.8-27B-TQ_S.gguf drafters/Qwen3.8-27B-DFlash2-TQ3.gguf mtp-vocab/atx_65536.txt --local-dir models
Run
These files need TQLLM: their TQ3 / TQ4 tensors use TQLLM's own GGUF types (300, 301), so stock llama.cpp, Ollama and LM Studio cannot load them.
docker run --rm --gpus all -p 127.0.0.1:8086:8086 -v "$PWD/models:/models:ro" \
-e MODEL_FILE=/models/Qwen3.8-27B-TQ_S.gguf \
-e DRAFT_MODEL_FILE=/models/drafters/Qwen3.8-27B-DFlash2-TQ3.gguf \
-e DRAFT_VOCAB_FILE=/models/mtp-vocab/atx_65536.txt \
-e KV_FORMAT=tq3 -e CONTEXT=32768 -e REASONING=on -e MAX_TOKENS=4096 \
ytan2000/tqllm:beta-5ae1388-cu13
The image runs on x86-64 (RTX 30 / 40 / 50) and arm64 (GB10 / DGX Spark). The server is OpenAI-compatible
(/v1/chat/completions). All options: docker run --rm --gpus all --entrypoint tqserve ytan2000/tqllm:beta-5ae1388-cu13 --help.
- Drafter: the draft model proposes several tokens and the 27B checks them in one pass; every accepted token is exact, so the output is the model's own text, only faster.
- Long context:
KV_FORMAT=tq3stores the KV cache at about 3 bits per value. On 16 GB cards TqCache keeps the oldest KV in system RAM when the requested context does not fit on the GPU (exact, at a decode cost from PCIe).
Quality
| TQ_S | |
|---|---|
| KLD vs BF16 (mean per-token KL divergence, lower is closer) | 0.055 |
| Same top token as BF16 | 90.1 % |
| GSM8K (1319, greedy, thinking on) | 94.6 % |
| MMLU-Pro (291-question subset) | 83.2 % |
| GPQA Diamond (198) | 82.83 % |
| BenchLoop overall | 87.5 |
GSM8K / MMLU-Pro on GB10, GPQA / BenchLoop on RTX 3090, TQLLM release engine.
Speed (LocalMaxxing canonical prompts, DFlash2-TQ3 drafter, 32K context, thinking on, median)
| Hardware | reasoning-v1 tok/s | code-v1 tok/s |
|---|---|---|
| RTX 3090 24 GB | 251.7 | 173.4 |
| RTX 5060 Ti 16 GB (Windows 11 / WSL 2) | 166.5 | 109.0 |
| GB10 (DGX Spark) | 92.8 | 60.4 |
Decode speed with a drafter depends on the text: code and copy-heavy answers accept more drafts than open prose.
Format
TQ3 (3.125 bits per weight) and TQ4 (4.125) rotate each 128-weight block and store one FP16 scale per block plus 8- or 16-level codes; the rotation spreads outliers so one scale per 128 weights is enough. For comparison, TQ3_4S (the format of YTan2000/Qwen3.8-27B-TQ3_4S) uses 32-weight blocks with four 8-bit scales, 4.0 bits per weight.
Chat template
The file embeds the fixed Qwen chat template; no --chat-template-file is needed.