gemma-4-26B-A4B-it — Blackwell-Optimized GGUF

Hand-tuned mixed-precision quantization for Gemma-4 26B-A4B.
13.1 GB · 4.50 BPW · Wikitext-2 PPL 226.42 ±3.35

Tensor types and layer zones were manually designed and iterated across 20+ experiments specifically for the RTX 5080's Blackwell MXFP4 MMA tiles. All routed expert tensors use MXFP4 — the only format with a dedicated sm_120 hardware path. Always-active weights (dense FFN, attention, embeddings) use higher-precision formats tuned per-role.

Files

Quick Start (RTX 5080)

llama-server \
  -m gemma-4-26B-A4B-it.gguf \
  -ngl 99 \
  -c 165376 \
  -np 1 \
  -ctk mxfp4 -ctv mxfp4 \
  --jinja \
  --temp 1.0 \
  --top-k 64 \
  --top-p 0.95 \
  --chat-template-kwargs '{"enable_thinking":true}' \
  --prio 3 \
  --host 0.0.0.0 --port 8080

Context size: 165,376 tokens (322×512, verified to fit within 15,558 MiB VRAM on RTX 5080 with mxfp4 KV cache).

Without mxfp4 KV cache (mainline llama.cpp), reduce to -c 97792 and use -ctk q8_0 -ctv q8_0 instead.

RTX 5080 / Blackwell Requirements

1. llama.cpp build — MXFP4 weights (mainline)

MXFP4 weight inference is available in llama.cpp b8493 and later. Build with the 120f-real architecture target — this is required for MXFP4 PTX instructions; plain 120 will fail (issue #19662):

cmake -B build \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES="120f-real" \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda-13.2/bin/nvcc \
  -G Ninja

cmake --build build --config Release -j $(nproc)

120f vs 120: 120f-real enables the full Blackwell instruction set including block-scaled MMA for MXFP4. Without the f suffix, MXFP4 ops fall back to software emulation or fail outright.

2. MXFP4 KV cache (fork, optional — unlocks full context)

The -ctk mxfp4 -ctv mxfp4 flags require the feat/mxfp4-kv-cache branch, rebased on b8675 (commit 06ffd4dfd). This is not yet merged to mainline.

git clone https://github.com/lucaspirola/llama.cpp.git
cd llama.cpp
# build as above with 120f-real

Without mxfp4 KV cache, you can still run this model at reduced context with -ctk q8_0 -ctv q8_0.

3. CUDA Toolkit

Requires CUDA 12.8+ or CUDA 13.2. On CUDA 12.6 or earlier, sm_120 codegen is unavailable.

VRAM budget (RTX 5080, 16 GB)

Context is limited by VRAM, not architecture. Gemma-4 supports up to 128K (SWA layers) / unlimited (FULL layers) natively.

Architecture

Weight Types

MXFP4 attention breaks Gemma-4: despite valid dimensions, MXFP4's fixed-grid block scaling crushes attention outliers (verified across 6 failed experiments, PPL 375–140k). All attention kept at Q5_K.

Layer zones (30 layers)

All zones use MXFP4 for experts — the format is both the most compressed and the fastest on Blackwell sm_120.

Perplexity Results (Wikitext-2)

All rows measured with llama-perplexity on the same hardware and llama.cpp build (576 chunks, n_ctx=512). Lower is better.

This quant beats mudler's APEX I-Compact by 1.65 PPL points while being 1.9 GB smaller.

Imatrix

Calibrated with unsloth's imatrix (imatrix_unsloth.gguf_file) — diverse dataset covering chat, code, reasoning, tool-calling, and agentic traces.

Reproducibility

The quantization config was hand-designed; mudler/apex-quant was used to generate the tensor-type file and invoke llama-quantize.

# Tensor-type config (hand-tuned settings)
./apex-quant/scripts/gen_config_gemma4.sh \
  --layers 30 \
  --edge-exp MXFP4 --near-exp MXFP4 --mid-exp MXFP4 \
  --edge-exp-down MXFP4 --near-exp-down MXFP4 --mid-exp-down MXFP4 \
  --dense Q6_K --dense-down Q8_0 \
  --attn Q5_K \
  --embd Q6_K --output Q6_K \
  -o config.txt

# Quantize
LLAMA_CPP_DIR=/path/to/llama.cpp/build/bin \
./apex-quant/scripts/quantize.sh \
  --config config.txt \
  --imatrix imatrix_unsloth.gguf_file \
  --base-type Q8_0 \
  input-bf16.gguf output.gguf

License

Gemma Terms of Use — see google/gemma-4-26b-a4b-it.