gemma4-31b-imatrix-mtp-GGUF

📊 Unified benchmark & quality table

Agentic metrics from a SWE-rebench holdout run through the OpenAI Agents SDK (10 instances × 3 reps). Static metrics (PPL / KLD / top-p) measured against FP16 on a held-out eval corpus at ctx=4096. KLD column is median for robustness to per-token tails.

† IQ2_M now ships an AWQ build (see § Why the 2-bit is AWQ below). The SWE-rebench figures (†) were measured on the earlier imatrix 2-bit; the AWQ swap is justified by a direct tool-call fidelity comparison (param-acc +54%, far tighter run-to-run variance) at the same 2.85 bpw. AWQ also cuts IQ2_M perplexity nearly in half (1959 → 1040) — it trades a hair of median-KLD for a much more usable 2-bit.

Q5_K_S resolves 40% of the holdout (tying the old IQ2_M, ahead of IQ3_M) at 100% patch and a low 11% tool-error rate; IQ4_XS remains the agentic leader at 47% (gap within run-to-run noise).

Sampling: temperature=0.25, top_p=0.95, top_k=20, max_tokens=32768, ctx=131072, thinking=false. Run on Apple Silicon (Metal); SWE-rebench linux/amd64 images under emulation, so wall-clock is relative, not absolute.

Pass Rate = gold tests pass after agent's patch (real resolution). Patch Rate = non-empty diff produced.

🧪 Why the 2-bit is AWQ

Static KLD/PPL measures how closely a quant's logits track FP16 — it does not measure whether the model still fills correct tool-call arguments, which is what agentic use actually needs. At 2 bits the plain-imatrix build keeps decent tool selection but loses argument precision. We replay 25 held-out real tool-use sessions (disjoint from calibration) through llama-server, scoring per assistant turn whether the model picks the right tool (tool-sel) and fills the right arguments (param-acc). 3 reps, mean ± σ, same sampling as above.

Both builds are measured on the same held-out eval + FP16 baseline (static) and the same 25-session replay (agentic). Static: ↓ PPL / ↓ KLD / ↑ top_p is better. Agentic: ↑ is better.

Read the split: on the metric the static table headlines (median-KLD) imatrix looks better (1.571 vs 1.804), and it edges top_p (46.6% vs 43.9%) — yet the AWQ build is the one that actually tool-calls. AWQ activation-aware scaling wins where it counts: +54% tool-argument accuracy (0.17 → 0.26) and — just as important for agents — it collapses run-to-run variance (param-acc ±.009 vs ±.082; the imatrix build swings 0.51/0.48/0.38 on tool-sel across seeds, AWQ holds 0.50/0.50/0.48). (AWQ also nearly halves PPL, 1959 → 1040 — PPL and KLD disagree here because the fold shifts the whole distribution, lowering average surprise while nudging the very top token slightly.) The upshot: the static table alone would have picked the wrong build. All 60 gemma layers are standard attention, so AWQ scales the full network (120 groups); the per-channel scales fold into RMSNorm with no inference cost. The higher-bit builds (IQ3_M+) don't need it — at 3-4 bits the imatrix build already preserves arguments and AWQ's fold slightly perturbs general text, so those ship plain imatrix.

Harness: quant-tuner run_toolcall_reps.py. This lighter replay is the proxy used to rank the 2-bit builds; a full SWE-rebench re-run on the AWQ build is pending (the table's IQ2_M SWE-rebench figures are the prior imatrix build's).

🔬 How it was made

• Hybrid imatrix — activation energy E[a²] mixed with weight-column energy ‖W[:,i]‖²·E[a²] per tensor, collected over real coding/tool-use logs + wiki.test.raw via quant-tuner. Used for the IQ3_M / IQ4_XS / Q5_K_S builds.
• AWQ for the 2-bit — the IQ2_M build additionally applies activation-aware weight scaling: each input channel is rescaled by mean(|x|)^α (α grid-searched on the tool-use logs, sampled across the whole corpus) and the inverse folds into the preceding RMSNorm — mathematically identity in FP16, but it moves the channels that carry tool-argument signal onto finer quant levels. All 60 layers are standard attention, so the fold covers the full network; the imatrix is then collected on the folded weights. Net: +54% tool-argument accuracy vs plain 2-bit imatrix, at no inference cost. See § Why the 2-bit is AWQ.
• IQ2_M codebook — 2-bit E8-lattice non-uniform codes with per-tensor tier bumps (attention output, early ffn_down get more bits). llama-quantize decides the mix.
• Vision mmproj — the model's SigLIP-style vision tower (27 layers, 280 soft tokens/image) exported separately at Q8_0 with convert_hf_to_gguf.py --mmproj (visually lossless, 772 MB), so the encoder stays high-precision while the text path runs at 2 bits. No audio encoder is shipped (the source has none).
• Disjoint splits — calibration (imatrix), validation (per-tensor α gate), and eval (PPL/KLD) come from different corpora; the SWE-rebench holdout never appears in any calibration set.
• Toolchain: quant-tuner for imatrix calibration, llama.cpp @ f3e1828 for final quantization. Calibration logs mined with LogMiner.

🚀 Usage

Ollama

ollama run hf.co/pearsonkyle/gemma4-31b-imatrix-mtp-GGUF:IQ2_M

llama.cpp (GPU)

# Build with CUDA (-DGGML_CUDA=OFF for CPU/Metal)
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-server
cp llama.cpp/build/bin/llama-* llama.cpp/

# Run the server
./llama-server \
    --model gemma-4-31B-it-IQ2_M.gguf \
    --ctx-size 16384 --n-gpu-layers 999 --split-mode layer \
    --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 \
    --parallel 1 --batch-size 2048 --ubatch-size 512 \
    --host 0.0.0.0 --port 1234

OpenAI-compatible API (Python)

import json, urllib.request

def ask(content, max_tokens=256):
    body = {
        "messages": [{"role": "user", "content": content}],
        "max_tokens": max_tokens,
        # Gemma 4 is a thinking model — disable or raise max_tokens
        "chat_template_kwargs": {"enable_thinking": False},
    }
    req = urllib.request.Request(
        "http://127.0.0.1:1234/v1/chat/completions",
        json.dumps(body).encode(),
        {"Content-Type": "application/json"},
    )
    return json.loads(urllib.request.urlopen(req).read())["choices"][0]["message"]["content"]

print(ask("What is 1+1?"))

🖼️ Vision (text + image)

Gemma 4 is natively multimodal. The vision tower ships separately as mmproj-gemma-4-31B-it-Q8_0.gguf (772 MB) so you only download it if you need images. It pairs with any of the four quant files (IQ2_M / IQ3_M / IQ4_XS / Q5_K_S) — the text weights are identical; the mmproj just adds the SigLIP encoder + projector.

One-shot from the CLI (llama-mtmd-cli):

./llama-mtmd-cli \
    --model gemma-4-31B-it-IQ4_XS.gguf \
    --mmproj mmproj-gemma-4-31B-it-Q8_0.gguf \
    --image screenshot.png \
    --jinja -ngl 999 --temp 0.2 -n 256 \
    -p "Describe this image. What's in it?"

--jinja is required — Gemma 4's chat template is Jinja-based and the CLI aborts without it. --image can be repeated for multi-image prompts; URLs work too.

⚠️ Thinking + the CLI. Gemma 4 is a reasoning model. From llama-mtmd-cli, leave thinking on and give it enough budget (-n 800+) so the answer survives the reasoning preamble — the --chat-template-kwargs '{"enable_thinking":false}' flag currently returns an empty completion on the CLI path. To get a clean, reasoning-free answer, disable thinking over the HTTP server instead (below).

Vision server — host the quant with the mmproj attached (this is exactly how the worked example above was generated). --jinja is required; the vision tower is loaded via --mmproj:

./llama-server \
    -m gemma-4-31B-it-IQ4_XS.gguf \
    --mmproj mmproj-gemma-4-31B-it-Q8_0.gguf \
    --jinja --ctx-size 8192 --n-gpu-layers 999 \
    --host 127.0.0.1 --port 1234

Vision is purely additive — drop the --mmproj flag and you're back to the identical text-only model.

The OpenAI-compatible /v1/chat/completions endpoint then accepts image_url content parts. With chat_template_kwargs.enable_thinking=false the server returns just the answer (no reasoning preamble). This is the exact call used to generate the mecha prompt above:

import base64, json, urllib.request

with open("mecha.png", "rb") as f:
    b64 = base64.b64encode(f.read()).decode()

body = {
    "messages": [{"role": "user", "content": [
        {"type": "text", "text": (
            "Look at this image and write a single, detailed text-to-image "
            "generation prompt that would recreate it. Cover the subject, colors, "
            "pose, lighting, style, and background. Respond with only the prompt."
        )},
        {"type": "image_url",
         "image_url": {"url": f"data:image/png;base64,{b64}"}},
    ]}],
    "max_tokens": 400,
    "temperature": 0.3,
    "chat_template_kwargs": {"enable_thinking": False},
}
req = urllib.request.Request(
    "http://127.0.0.1:1234/v1/chat/completions",
    json.dumps(body).encode(),
    {"Content-Type": "application/json"},
)
print(json.loads(urllib.request.urlopen(req).read())["choices"][0]["message"]["content"])

Worked example — same image, all three quants

Each shipped quant was handed the mecha on the right and asked to "write a single, detailed text-to-image generation prompt that would recreate it".

⚡ Speculative decoding (MTP drafter)

This repo also bundles a multi-token-prediction (MTP) drafter at the repo root, mtp-gemma-4-31B-it.gguf (358 MB, Q4_K_M) — a self-quantized conversion of google/gemma-4-31B-it-assistant (arch gemma4-assistant, nextn_predict_layers = 4). The acceptance rates where the model's drafted tokens were accepted by the trunk were measured using three reps over 5 mixed coding/reasoning prompts × 200 tokens, temperature=0.3, thinking off; distinct seeds per rep.

Acceptance holds up across all four trunks — the highest-fidelity Q5_K_S leads at every draft depth (87.9% at n=1, still 66.0% at n=4), and even the 2-bit IQ2_M accepts 83% of single-token drafts.

How small can the drafter be? Fixing the trunk at IQ2_M and quantizing the drafter itself shows acceptance barely moves down to Q4_K_M, then erodes at 2-bit — and the erosion grows with draft depth (n):

Q6_K and Q4_K_M are statistically free — indistinguishable from the Q8_0 drafter at every depth (within ±1σ); the drafter only has to be directionally right for the trunk to accept.

Usage — add --model-draft + --spec-type draft-mtp to the server command:

./llama-server \
    -m gemma-4-31B-it-IQ4_XS.gguf \
    --model-draft mtp-gemma-4-31B-it.gguf \
    --spec-type draft-mtp --spec-draft-n-max 4 \
    --jinja -ngl 999 -fa on \
    --host 127.0.0.1 --port 1234

The drafter lives at the repo root so --spec-type draft-mtp auto-discovers it when you load the trunk with -hf (no manual --model-draft needed): llama-server -hf pearsonkyle/gemma4-31b-imatrix-mtp-GGUF:IQ4_XS --spec-type draft-mtp --spec-draft-n-max 4.

🪪 License & attribution

• Inherits the Gemma Terms of Use from the base model.
• Base weights: google/gemma-4-31B-it.
• MTP drafter converted from google/gemma-4-31B-it-assistant (same Gemma Terms of Use).
• Calibration + quantization: Quant-Tuner with vendored llama.cpp @ f3e1828.
• Calibration logs mined with LogMiner.