GRM-2.6-Plus-0628 — Reasoning-imatrix GGUF

Quantized GGUF builds of OrionLLM/GRM-2.6-Plus-0628 (27B agentic reasoning model, Qwen3.5/3.6 architecture with an MTP layer), calibrated with a reasoning-domain importance matrix rather than a generic one, and with the recurrent DeltaNet path protected in every quant.

Update 2026-07-30 — chat template fix (v2)

Every GGUF in this repo has been re-issued with the corrected chat template, aligning with the base model's 2026-07-27 Update — Chat template fix (tool calls inside reasoning). This is the main reason to re-download.

The template shipped with the original weights breaks agentic loops in two ways:

• Tool calls land in reasoning_content instead of tool_calls. The client sees a model that talked about calling a function instead of one that called it, and the loop stalls.
• An assistant turn that merely quotes </think> in prose is truncated at that point, silently destroying the rest of the message.

These builds embed froggeric v21.3, which fixes both and removes a preamble directive that made the model hallucinate closing tags when reasoning was disabled. --jinja alone is enough; no --chat-template-file.

If you pulled these files before this update, re-download — or pass the template externally.

Quant table

Each file measured against the bf16 weights, not against another quant: KL divergence on a held-out reasoning corpus (48 × 512 tokens), -ngl 99 --flash-attn auto. Lower KLD and RMS Δp are better, higher top-1 is better. PPL wiki is wikitext-2 test, a different corpus, given for comparison with other repos. Sizes in GiB, to compare directly against VRAM.

The ladder is monotonic in bit width on all three KL metrics, with no inversions. The 99.0 % quantile is used rather than 99.9 % or the maximum: over 24 576 tokens those rest on roughly 25 tokens and on 1 respectively, which is not enough to separate builds.

Q4_K_XL = Unsloth-style per-tensor recipe + imatrix, from community feedback. It replays Unsloth's Dynamic (UD-Q4_K_XL) bit mixture on GRM, rebuilt on this reasoning imatrix. The MTP layer is capped at Q4_K: speculative-decoding acceptance saturates by Q4, so higher-precision MTP bits buy nothing.

DeltaNet protection

GRM's hybrid layers are gated DeltaNet. The output projection ssm_out funnels the whole recurrent branch back into the residual stream, and it is the tensor Unsloth's analysis flags as needing high precision. Every quant here lifts it above its tier's base precision, across all 48 DeltaNet layers, on a ladder scaled to the tier:

SSM gates (ssm_alpha, ssm_beta) are F32 throughout. Cost: +0.13 to +0.26 bpw, on one tensor per layer.

Perplexity barely sees this, because the damage lives in the tail. KL divergence against the bf16 weights does.

Single-variable ablation — ssm_out at the tier's base precision (control) versus protected (shipped), everything else identical: same imatrix, same corpus, same chunking. Seven tiers, two context lengths. Negative = the protected build is better.

• 99.0 % KLD improves in all fourteen cells, top-1 agreement in all fourteen, RMS Δp in thirteen.
• At Q5, perplexity and KL disagree outright. Wikitext PPL says the protected build is 0.15 % worse; its 99.0 % KLD is 20 % better, along with RMS Δp and top-1. The mean did not merely miss the effect, it pointed the wrong way.
• Q6_K gains the least — +0.03 pt of top-1 at best. It ships protected for recipe consistency across the ladder.
• Worst-case (Max KLD) is not reported. It moves from −76 % to +50 % with no ordering across tiers, because it rests on a single token.

Usage (llama.cpp)

The real 24 GB config — IQ4_XS at 262K context with vision.

llama-server \
  --model "$MODEL" \
  --mmproj "$MMPROJ" \
  --host 127.0.0.1 \
  --port 5000 \
  --ctx-size 262144 \
  --n-gpu-layers 65 \
  --cache-type-k q4_0 \
  --cache-type-v q4_0 \
  --batch-size 256 \
  --ubatch-size 256 \
  --rope-freq-base 10000000 \
  --parallel 2 \
  --kv-unified \
  --slots \
  --jinja \
  --flash-attn auto \
  --image-min-tokens 1024 \
  --cache-ram 8192 \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --spec-draft-n-min 1 \
  --no-mmproj-offload

--spec-type draft-mtp enables speculative decoding through the built-in MTP layer — same GGUF, no separate draft model.

Vision

To enable image input, pass a Qwen3.6-27B multimodal projector: DAXZEIT/Qwen3.6-27B-mmproj-hybrid-Q8_0-F16-gguf. Most tensors Q8_0, ffn_down kept at F16 because a full-Q8 down-weight does not hold up. ~601 MB on disk, ~629 MB of VRAM on GPU.

imatrix

Calibrated on a reasoning corpus (chain-of-thought, expository QA, math word problems), block-shuffled, 400 chunks × 512 tokens, statistics collected from Q6_K.

The importance matrix is a coarse per-channel activation statistic: what matters is which activations it samples, not raw token count. Calibrating on reasoning concentrates it on the channels this model uses for step-by-step work — and in controlled testing wikitext perplexity was on par with a generic calibration, so it is a targeted choice rather than a trade-off.

The imatrix file is included (grm-2.6-plus-0628-reasoning.imatrix, GGUF format) — build your own quants on this calibration instead of a generic one. Note for very-low-bit IQ builds: the blk.64 MTP layer is never exercised in a forward pass and collects zero imatrix entries, so IQ quants bail on it. Force it with --tensor-type blk.64.=q4_K.

Credits

• Base model: OrionLLM/GRM-2.6-Plus-0628 (Apache-2.0)
• Quantization: llama.cpp
• Chat template: froggeric v21.3
• Q4_K_XL per-tensor recipe adapted from Unsloth's Dynamic (UD) mixture; the ssm_out worst-case analysis motivating the DeltaNet protection is theirs
• Reasoning imatrix + builds: DAXZEIT