Qwen3.8-27B-imatrix-MTP-GGUF

📊 Unified benchmark & quality table

• 📐 Static — measured against FP16 on the external holdout (code + math + tools, 90k tokens, disjoint from every calibration source) at eval-ctx 8192. KLD is median; top_p is top-token agreement with FP16.
• 🤖 Agentic — 25 held-out tool-use sessions (174 scored turns), plus one SWE-rebench issue solved end-to-end through the OpenAI Agents SDK.

⚠️ The 🤖 rows are a band, not a ranking. At n≈174 turns the standard error is ±3.8pp, and the whole spread — FP16 included — is 0.454–0.503. That is a 4.9pp range inside one standard error. It supports "every rung, down to 3.06 bpw, tool-calls indistinguishably from FP16 here" and nothing finer. IQ2_M and FP16 both score 86/174 yet disagree on 24 turns (12 each way) — the tie is a coincidence, not equivalence.

📐 KLD and top_p are the trustworthy columns, strictly monotone across all six evals. IQ2_M gives up ~28% of top-token decisions — a memory-constrained option, not a drop-in.

Sampling. Static: eval-ctx 8192. Tool-call replay: greedy (temperature=0), ctx=32768, --no-stop-on-fail so every model is scored on the identical 174 turns (the default stop-on-fail halts a weak model early and scores it on fewer, easier turns — which makes models incomparable). Agentic: temperature=0.25, top_p=0.95, max_tokens=8096, 2,048-token reasoning budget, step cap 60. Run on an RTX PRO 6000 Blackwell.

"Tool errors" are classified, not counted. A non-zero exit is not an agent error — a grep that legitimately matched nothing and a pytest correctly reporting a failure both exit non-zero. Errors are split into malformed (shell syntax error, command not found, failed cd — the intended work never ran), timeout, and nonzero (the program ran and reported a result). Under that split every rung issues zero malformed commands; the only genuine tool failure in the whole sweep is one timeout on IQ3_M.

The reasoning budget is a cost control, not a quality lever. Uncapped, IQ2_M was observed emitting a 12,566-token single completion — a repetition loop, not reasoning. An unbudgeted control sweep was run alongside: at n=1 the 2k budget shows no reliable effect on whether the issue gets solved (IQ4_XS and Q5_K_M improved, IQ3_M got worse, FP16 rose).

Six eval distributions, each with its own FP16 baseline — never concatenated:

⚠️ The four chat-templated evals (tools, agentic, broad, cal8k) are quant-vs-quant only: llama-perplexity has no --parse-special, so chat markers tokenize as ordinary BPE and absolute PPL is off-distribution. KLD and top_p remain valid — which is why PPL goes non-monotone there while KLD does not.

📚 What it was calibrated on

An imatrix is only as good as the text it watched. Here is exactly what this one saw.

Three things worth reading off that table:

• Nearly two-thirds is agentic (logs + swe-trajectories = 63%), and those are the long windows — median 13,004 and 9,650 tokens against a 182-token median for broad-instruct. 84 of the 125 log windows exceed 7,500 tokens and 50 exceed 16,384, which is only possible because packing is at ctx 32,768. Nothing was truncated: 0 sessions hit the 32,076-token cap.
• Tool-calling is dense, not decorative — 6,786 <tool_call> blocks and 5,827 <tool_response> blocks across 76 distinct tool schemas, and in the log windows 99.3% of tokens sit in tool-bearing turns.
• reasoning overlaps logs by design. It re-renders the same conversations with a reasoning turn placed last, because chat templates keep <think> only on a render's final assistant turn and scrub it from history. Without those extra windows the corpus would contain almost no reasoning at all.

The red-team slice ships attack prompts with generic refusals substituted — the targets' original harmful completions never enter any corpus. Refusal behaviour is what low-bit quantization erodes first, so the attack distribution belongs in calibration; the responses do not.

🔬 How they were made

• Hybrid imatrix at ctx 32,768 — activation energy E[a²] blended with weight-column energy ‖W[:,c]‖²·E[a²] per tensor, collected over the corpus above in 129 chunks of a full 32,768-token context (3h54m on an RTX PRO 6000 Blackwell). 496 tensors received statistics.
• SSM passthrough — Qwen3.8 is hybrid (48 linear-attention + 16 full-attention layers). 144 SSM tensors pass through with raw E[a²]; output-aware re-ranking is mathematically invalid for state-space tensors.
• Bundled MTP draft head — the trained MTP head (mtp_num_hidden_layers=1) is remapped to blk.64 and pinned to Q8_0 in every rung: 8 × Q8_0 + 7 × F32, zero low-bit tensors, verified per file. This is checked rather than assumed because llama-quantize silently accepts a --tensor-type pattern that matches nothing — a stale pin would quantize the draft head with the trunk and surface only as poor acceptance. The head gets no imatrix statistics; it sits outside the forward pass, which is why it is pinned rather than calibrated.
• The tool-call format trap — Qwen3.8's chat template emits tool calls as XML (<tool_call><function=NAME><parameter=KEY>), not the JSON form earlier Qwen3.x templates used. Calibrating on the JSON form would have collected statistics on a syntax this model never produces.
• output.weight is calibrated — llama-imatrix only collects tensors named blk.* unless you pass --process-output, so the largest quantized tensor (and the one that most directly shapes the distribution KLD measures) is otherwise quantized blind. llama-quantize prints did not find weights for output.weight and proceeds anyway, so the only symptom is a line in a log.
• Disjoint splits — calibration (train) and every eval holdout are disjoint by construction, so the static table measures generalization, not fit. The one exception is cal8k, explicitly labelled a fit probe.
• Toolchain: Quant-Tuner with vendored llama.cpp @ f3e1828. Calibration logs mined with LogMiner.

git clone https://github.com/pearsonkyle/Quant-Tuner && cd Quant-Tuner
git submodule update --init --recursive
cmake -S vendor/llama.cpp -B vendor/llama.cpp/build -DGGML_CUDA=ON \
      -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=120 -DLLAMA_CURL=ON
cmake --build vendor/llama.cpp/build -j 64
uv sync --extra dev

PYTHONPATH=src .venv/bin/python scripts/exp060_repack_cal_32k.py --ctx 32768 \
    --wiki out/exp-001/wiki/wiki.test.raw
PYTHONPATH=src .venv/bin/python scripts/exp060_quants_qwen38.py \
    --run exp-060-32k --ctx 32768 --eval-ctx 8192 \
    --evals external general tools agentic broad cal8k

⚠️ llama-imatrix needs --no-ppl above ~17k ctx on this model. Its perplexity path computes all_logits + first*n_vocab with first = n_ctx/2 in int arithmetic (tools/imatrix/imatrix.cpp:911). With Qwen3.8's 248,320-token vocab that product overflows INT_MAX for any n_ctx > 2³²/248320 ≈ 17,296, and the process segfaults after the first pass. --no-ppl skips only the perplexity bookkeeping — the forward pass, and therefore every activation statistic, is unchanged.

🚀 Usage

Ollama

ollama run hf.co/pearsonkyle/Qwen3.8-27B-imatrix-MTP-GGUF:IQ4_XS
# also: :Q5_K_M · :IQ3_M · :IQ2_M

llama.cpp server, with MTP speculative decoding

./llama-server \
    --model Qwen3.8-27B-IQ4_XS.gguf \
    --ctx-size 32768 --n-gpu-layers 999 \
    --spec-type draft-mtp --spec-draft-n-max 1 \
    --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 \
    --host 0.0.0.0 --port 1234

The draft head is inside the GGUF — no --model-draft, nothing extra to fetch. Qwen3.8 exposes one nextn layer, so --spec-draft-n-max 1 is the right setting. Drop both --spec-* flags to run without speculative decoding.

🖼️ Vision (text + image)

The vision tower ships separately as mmproj-Qwen3.8-27B-Q8_0.gguf (629 MB) so you only download it if you need images. It pairs with any of the four rungs — the text weights are identical; the mmproj just adds the encoder + projector.

# one-shot from the CLI
./llama-mtmd-cli \
    -m Qwen3.8-27B-IQ4_XS.gguf \
    --mmproj mmproj-Qwen3.8-27B-Q8_0.gguf \
    --image screenshot.png \
    --jinja -ngl 999 --temp 0.2 -n 512 \
    -p "Describe this image."

# or serve it — /v1/chat/completions then accepts image_url content parts
./llama-server \
    -m Qwen3.8-27B-IQ4_XS.gguf \
    --mmproj mmproj-Qwen3.8-27B-Q8_0.gguf \
    --jinja --ctx-size 32768 --n-gpu-layers 999 \
    --host 0.0.0.0 --port 1234

--jinja is required — the chat template is Jinja-based and the multimodal path needs it. For grounding tasks (pointing at or locating things in an image) llama.cpp recommends --image-min-tokens 1024; the default tokenization is fine for description and Q&A.

OpenAI-compatible API

import json, urllib.request

def ask(content, max_tokens=512):
    body = {"messages": [{"role": "user", "content": content}], "max_tokens": max_tokens}
    req = urllib.request.Request("http://127.0.0.1:1234/v1/chat/completions",
                                 json.dumps(body).encode(),
                                 {"Content-Type": "application/json"})
    return json.loads(urllib.request.urlopen(req).read())["choices"][0]["message"]["content"]

print(ask("Write a Python function that reverses a linked list."))

🪪 License & attribution

• Inherits the license of the base model Qwen/Qwen3.8-27B (Apache-2.0).
• Calibration + quantization: Quant-Tuner with vendored llama.cpp @ f3e1828.
• Calibration logs mined with LogMiner.