Gemma-4-31B-IT NVFP4 GGUF
NVFP4 GGUF quantizations of google/gemma-4-31B-it, produced for use with llama.cpp.
The dense FFN tensors are quantized to NVFP4 (NVIDIA's 4-bit float with E4M3 block scale), repacked from the official nvidia/Gemma-4-31B-IT-NVFP4 ModelOpt calibration. The remaining tensors (attention projections, embeddings, output) use a conventional GGUF quant — three variants are provided.
About LibertAI
LibertAI is a decentralized AI platform — private inference, an OpenAI-compatible API, and a chat UI, all running on community GPUs over Aleph Cloud instead of a single company's servers. No accounts required to chat, no logs sent home, and the same models you'd self-host are available behind a sovereign endpoint.
If you want to put this model (or any other) to work as an autonomous agent without running your own infrastructure, check out LiberClaw — Hermes-style agents hosted on Aleph Cloud with LibertAI inference. Free tier: 2 agents, no credit card, 5 minutes to deploy. Open source.
Why NVFP4? On NVIDIA Blackwell GPUs (RTX 50-series, B100/B200), llama.cpp uses native NVFP4 tensor-core MMA kernels (added in llama.cpp #22196) for the FFN matmul — the dominant compute cost during inference. On older GPUs the path falls back to
dp4a/MMQ kernels, where these GGUFs run but offer no perf advantage over standard K-quants.
Files
| File | Size | FFN | Other tensors | When to pick |
|---|---|---|---|---|
Gemma-4-31B-IT-NVFP4-Q4_K_M.gguf | 17 GB | NVFP4 | Q4_K_M | Recommended. Fastest serving throughput on Blackwell + smallest VRAM footprint |
Gemma-4-31B-IT-NVFP4-Q8_0.gguf | 21 GB | NVFP4 | Q8_0 | Higher precision attention/embeddings if you have the VRAM |
Gemma-4-31B-IT-NVFP4-BF16.gguf | 30 GB | NVFP4 | BF16 | Max quality (preserves source precision); slower in practice — only pick for bit-for-bit source fidelity |
mmproj-Gemma-4-31B-IT-F16.gguf | 1.2 GB | — | F16 vision + audio towers | Required for image/audio input — reusable with any Gemma-4-31B-IT GGUF |
Performance
Measured on an NVIDIA RTX 5090 (32 GB, Blackwell, sm_120), llama.cpp build c84e6d6db.
Batched serving (llama-batched-bench, 512 in / 128 out per request)
NVFP4-Q4_K_M beats stock Q4_K_M on total serving throughput at every parallel batch size we tested (+7% / +0% / +2% at 1 / 4 / 8 sequences), with consistent token-generation wins (+8% / +4% / +6%). It also uses less VRAM (16.6 vs 18.2 GiB), leaving more room for KV cache.
Gemma-4's unified-KV global attention has a large KV cache footprint — at parallel=16 with 8 K context the model + cache exceeds 32 GB on a 5090. If you serve high concurrency, the smaller weight footprint of the NVFP4 variant matters even more.
Variant comparison (same hardware, single-stream)
| Variant | Size | PP512 (tok/s) | TG64 (tok/s) |
|---|---|---|---|
NVFP4-Q4_K_M | 16.56 GiB | 2703 | 64.7 |
NVFP4-Q8_0 | 20.69 GiB | 2712 | 55.8 |
Usage
Text-only (CLI)
llama-cli -m Gemma-4-31B-IT-NVFP4-Q4_K_M.gguf -ngl 999 -c 8192 -p "Your prompt here"
Multimodal (server, image + audio + text)
llama-server \
-m Gemma-4-31B-IT-NVFP4-Q4_K_M.gguf \
--mmproj mmproj-Gemma-4-31B-IT-F16.gguf \
-ngl 999 -c 32768 \
--host 0.0.0.0 --port 8080
Gemma 4's mmproj bundles both vision and audio towers, so you can send image and audio content blocks — see the llama.cpp multimodal docs.
About the architecture
Gemma-4-31B-IT is a 30.7B-parameter dense decoder with a hybrid local-sliding + global attention scheme (unified KV in global layers, Proportional RoPE for long context). The NVIDIA NVFP4 source quantizes the 60 layers × 3 dense FFN projections (180 NVFP4 tensors total). The attention projections (attn_q/attn_output are particularly large in this architecture due to the unified-KV global attention) and embeddings stay at higher precision in the source — that's why the -BF16 variant is comparatively large at 30 GB.
Sources & credits
- Base model: google/gemma-4-31B-it by Google — Gemma license
- NVFP4 calibration source: official nvidia/Gemma-4-31B-IT-NVFP4 (NVIDIA ModelOpt v0.37)
- mmproj source: official BF16 weights from
google/gemma-4-31B-it - Tooling: llama.cpp
convert_hf_to_gguf.pyandllama-quantize
License
Inherits the Gemma license from the upstream model.