Gemma 4 31B Uncensored — Q4_1_G64 4.5 bpw for GP100

Q4_1_G64 · 4.52 bpw · 16.2 GiB · +59% vs Q4_K_M MTP on 2× P100

GGUF pack of HauhauCS/Gemma4-31B-QAT-Uncensored-HauhauCS-Balanced-MTP requantized to type-43 Q4_1_G64 for 2× Tesla P100 (sm_60).

This is the dense 31B. That matters: G64’s fast A16 / HFMA2 path needs ne[2]==1. All 410 body tensors here qualify. The 26B-A4B MoE does not — its 60 expert tensors are ne[2]=128 and fall through to dequant+cuBLAS (32 tok/s bare / 38 with MTP). Do not G64 that MoE and expect this result.

The parent Q4_K_M and the BF16 mmproj stay on the Hauhau repo. Stock llama.cpp cannot load type 43.

Why G64 exists (GP100 vs GP104, Q4_K tax, closed levers): DESIGN.md on the Qwen pack. How to G64 any model: CONVERT.md.

Files

G64 mix: 410× Q4_1_G64 + Q4_1 token_embd (tied, no separate output.weight) + 422× F32. Requant from Hauhau Q4_K_M, not F16 — second-generation quant. llama-quantize --pure --allow-requantize --output-tensor-type q8_0 --token-embedding-type q4_1 Q4_1_G64 (59.2 s). 17806 → 16556 MiB.

MTP is the official Gemma 4 31B assistant Hauhau ships (gemma4-assistant, 4 layers, embd 1024 / out 5376, nextn_predict_layers=4, ctx 131072).

Parent Q4_K_M (17.4 GiB) and BF16 mmproj (1.12 GiB): Hauhau card. The mmproj OOMs CUDA0 on 2×16 GB after 31B + 128K Q8 + MTP (~1145 MiB alloc). This pack is text-only at that window.

Speed (2× Tesla P100)

Same canonical prompt, 256 gen, temp 0, --allow-eos, GPU warm, hash-matched request 2+. Image llama-p100:pr-sm60-gmeta (needs the Gemma MTP shared-KV meta-backend fix). Width 2, BACKEND_SAMPLING=off, tensor-split 1,1, Q8 KV, text-only, ctx 131072.

G64 is +59% vs the Q4_K_M parent on the same draft. Accept stays in the same band. That is the kernel, not a luckier draft.

26B-A4B MoE Q4_K_M + official MTP is still faster in absolute tok/s (56.78 / 58.12, accept 0.681) because only ~4B parameters are active. This 31B pack is the dense model that can actually use G64.

Do not quote a 40 tok/s bare number from this table. These rows include MTP.

Run

Needs thefallentree/llama.cpp-gp100 with the Gemma MTP meta-backend fix (on cursor/gemma4-mtp-host-6006 as 418fc582d, tip 958bce0bb after the 2026-09-16 rebase onto 930e2fa59). ./run.sh gemma uses pr-sm60-gmeta. Do not serve this file on the Qwen LoopSpec image, and do not serve Qwen LoopSpec on the Gemma-MTP image. a8c9f3a7b / g64slot is archive only.

sm_60 only. P40 / GTX 10-series are sm_61; HFMA2 is gated off there.

./build/bin/llama-server \
  --model Gemma4-31B-QAT-Uncensored-HauhauCS-Balanced-Q4_1_G64-Q4_1-embd.gguf \
  --model-draft mtp-gemma-4-31B-it.gguf \
  --jinja \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 --spec-draft-n-min 0 --spec-draft-p-min 0 \
  --device CUDA0,CUDA1 --split-mode tensor --tensor-split 1,1 \
  --device-draft CUDA0,CUDA1 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --cache-type-k-draft q8_0 --cache-type-v-draft q8_0 \
  --ctx-size 131072 --flash-attn on

No --backend-sampling: tensor-split Gemma 500s (vector underflow). Leave GPU sampling off.

Architecture

gemma4, 31B dense, 60 layers, d_model 5376, vocab 262144, context 262144, sliding window 1024. Tied embeddings. Official MTP head is a separate gemma4-assistant GGUF.

Provenance

License

Gemma, inherited from google/gemma-4-31B-it. Local inference and research. Uncensored by HauhauCS — not a safety-tuned deployment model.