Qwen3.8-27B-TQ3_4S

TurboQuant TQ3_4S build of Qwen3.8-27B — the hybrid Gated DeltaNet + Gated Attention architecture with trained Multi-Token Prediction (MTP) heads.

Required Runtime

This model requires the TurboQuant runtime fork: turbo-tan/llama.cpp-tq3

Stock llama.cpp builds cannot load the TQ3_4S tensor type. This is an MTP model: the GGUF contains a trained MTP draft block (nextn heads) — serve it with draft-MTP speculative decoding for full speed.

Files & Versions

The mmproj files are the native vision-language projectors from unsloth/Qwen3.8-27B-GGUF — add --mmproj mmproj-BF16.gguf (or -F16) for image/video understanding.

v2 embeds the froggeric v22 fixed chat template directly in the GGUF — llama.cpp applies it automatically, no --chat-template-file needed. It fixes two bugs in the official 3.8 template:

1. String tool-arguments crash — the official template throws TypeError: Can only get item pairs from a mapping when clients send standard OpenAI-format string arguments (e.g. most agents/harnesses). v2 handles both dicts and JSON strings.
2. Forced xhigh reasoning steering — the official template injects "think carefully" steering instructions into every system prompt by default; v2 suppresses steering when thinking is off.

Same weights, same quant, same MTP heads — only the template differs. Measured impact: toolcall suite 81.7 → 86.7 (+5pp). If you use v1, add --chat-template-file chat_template.jinja (v22, included in this repo) for correct tool calling.

Model Details

• Parent model: Qwen/Qwen3.8-27B (quantized from the unsloth/Qwen3.8-27B-GGUF BF16 source)
• File: Qwen3.8-27B-TQ3_4S.gguf — 13.8 GB (4.24 BPW)
• Recipe: TQ3_4S with output layers + token embeddings at q6_K (the "out6k" recipe)
• Architecture: qwen3_5 — 64 layers, 48× Gated DeltaNet (linear attention) + 16× Gated Attention, trained multi-step MTP heads
• Context: 262,144 tokens native

Recommended Launch

./build/bin/llama-server \
  -m Qwen3.8-27B-TQ3_4S.gguf \
  --host 127.0.0.1 --port 8080 \
  -c 32768 -np 1 -ngl 99 -fa on \
  -ctk q8_0 -ctv tq3_0 \
  --spec-type draft-mtp \
  --spec-draft-n-min 1 \
  --spec-draft-n-max 2 \
  --spec-draft-p-min 0.0 \
  --no-spec-draft-backend-sampling \
  --reasoning off --jinja

Settings notes:

• --spec-draft-n-max 2 is optimal for this model's MTP head (measured)
• --no-spec-draft-backend-sampling keeps CUDA graph reuse high (~25% throughput difference)
• --reasoning off for agentic/benchmark work; the model supports thinking mode via --reasoning-format deepseek --reasoning-budget N

Reasoning-effort guidance (measured 2026-08-17/18, RTX 3090)

This hybrid Gated DeltaNet quant has an effort-dependent behavior worth knowing:

For long-reasoning tasks, cap the budget: --reasoning-budget 32768 (or lower). Uncapped xhigh can enter planning loops that never produce an artifact. Budget-capped thinking with a cut-off message (--reasoning-budget-message) keeps outputs converging.

Faster serving: 125 tok/s recipe (RTX 3090)

With turbo-tan/llama.cpp-tq3 main (>= 45efd44ec), the v2 file reaches 125.2 tok/s long-output decode (839-token prompt, 4096 generated, reasoning off) with the full 262K context: DFlash2 drafter + q4_0 KV cache. Built-in MTP chain 3 + q4_0 KV reaches 107.9 tok/s. Full launch scripts, before/after chart and raw runs: turbo-tan/recipes: qwen3.8-27b-tq3_4s-mtp-rtx3090.

MTP draft vocabulary (draft-vocab-40960.txt, in this repo): restricts MTP draft sampling to 40,960 frequent tokens, which speeds up the MTP draft step. Use with draft-MTP only (it slowed the DFlash2 drafter):

LLAMA_SPEC_CHAIN=1 llama-server -m Qwen3.8-27B-TQ3_4S-v2.gguf -c 262144 -ngl 99 -fa on --jinja \
  -ctk q4_0 -ctv q4_0 --spec-type draft-mtp --spec-draft-n-max 3 \
  --spec-draft-vocab-map draft-vocab-40960.txt \
  --no-backend-sampling --spec-draft-backend-sampling

Credits: the draft vocabulary is HyperQwen's shipped list (syv-ai, Apache-2.0; see draft-vocab-40960.NOTICE), converted to llama.cpp's map format. DFlash2 is by z-lab: paper, code, GGUF drafter.

Benchmarks

Validated on NVIDIA RTX 3090 (24 GB), single stream, draft-MTP speculative decoding (--spec-draft-n-max 2).

Model size: 13.8 GB (4.24 BPW).

Decode speed

The 3.8's MTP head is better trained than the 3.6's (acceptance 0.849 vs 0.796), yielding ~2.7 tokens per decode step on the same hardware and recipe — that's the source of the speed uplift.

Comparison vs Qwen3.6-27B-MTP-TQ3_4S (same TQ3_4S recipe)

Notes on the comparison: the 3.6 column is the published model-card number (run under that model's own tuned serving config). The 3.8 column is a first-pass battery — Hard86, task suites and speed under reasoning-off; evalplus under budget-256. The 3.8 wins decisively on decode speed, MTP acceptance and reason-math; on the task suites the 3.6's tuned-config numbers remain the reference until the 3.8 battery is re-tuned.

Tested Hardware

• NVIDIA RTX 3090 (24 GB) — all numbers above

License

Subject to the Qwen3.8 base model license (Apache 2.0) and the runtime components used to run the GGUF.