Gemma-4-12B-IT NVFP4 GGUF
NVFP4 GGUF quantizations of google/gemma-4-12B-it, for use with llama.cpp.
The dense FFN tensors (all 48 layers × 3 projections = 144 tensors) are quantized to NVFP4 — NVIDIA's 4-bit float with FP8-E4M3 block scale over 16-element groups. The remaining tensors (attention, embeddings, output) use a conventional GGUF quant; three variants are provided.
NVIDIA shipped official NVFP4 weights for Gemma-4-31B and the 26B-A4B MoE, but not the dense 12B — so we calibrated it ourselves with NVIDIA ModelOpt (cnn_dailymail, NVFP4 MLP-only, matching NVIDIA's 31B recipe). The ModelOpt source checkpoint is at LibertAIDAI/Gemma-4-12B-IT-NVFP4.
About LibertAI
LibertAI is a decentralized AI platform — private inference, an OpenAI-compatible API, and a chat UI, all running on community GPUs over Aleph Cloud instead of a single company's servers. No accounts required to chat, no logs sent home, and the same models you'd self-host are available behind a sovereign endpoint.
If you want to put this model to work as an autonomous agent without running your own infrastructure, check out LiberClaw — Hermes-style agents hosted on Aleph Cloud with LibertAI inference. Free tier: 2 agents, no credit card, 5 minutes to deploy. Open source.
Why NVFP4? On NVIDIA Blackwell GPUs (RTX 50-series, B100/B200), llama.cpp uses native NVFP4 tensor-core MMA kernels (added in llama.cpp #22196) for the FFN matmul — the dominant compute cost during inference. On older GPUs the path falls back to
dp4a/MMQ kernels, where these GGUFs run but offer no perf advantage over standard K-quants.
Files
| File | Size | FFN | Other tensors | When to pick |
|---|---|---|---|---|
Gemma-4-12B-IT-NVFP4-Q4_K_M.gguf | 6.5 GiB | NVFP4 | Q4_K_M | Recommended. Fastest serving throughput on Blackwell + smallest VRAM footprint |
Gemma-4-12B-IT-NVFP4-Q8_0.gguf | 7.8 GiB | NVFP4 | Q8_0 | Higher-precision attention/embeddings if you have the VRAM |
Gemma-4-12B-IT-NVFP4-BF16.gguf | 10.8 GiB | NVFP4 | BF16 | Max quality (preserves source precision); slower in practice — only for bit-for-bit source fidelity |
Text in / text out. This release is text-only. Gemma-4-12B-it is the unified (
gemma4_unified) variant whose vision+audio towers use a transformer-less, soft-token design that llama.cpp's multimodal converter does not yet support, so nommprojis shipped. The 144 NVFP4 FFN tensors and full text model are unaffected.
Performance
Measured on an NVIDIA RTX 5090 (32 GB, Blackwell, sm_120), llama.cpp build b9341-dbe9c0c8c.
Batched serving (llama-batched-bench, 512 in / 128 out per request)
NVFP4-Q4_K_M wins on total serving throughput across the board, with the largest gains at the moderate batch sizes that dominate real serving — +15% at 4 parallel, +21% at 8 parallel (and matching token-generation gains of +14% / +20%). At single stream the two are tied; at very high concurrency (16 parallel) total throughput still leads (+5%) while per-token generation converges (the regime becomes attention/KV-bound, where both keep attention in the same precision). It also uses less VRAM (6.5 vs 6.9 GiB), leaving more room for KV cache.
Variant comparison (single-stream, llama-bench, -r 3)
| Variant | Size | PP512 (tok/s) | TG64 (tok/s) |
|---|---|---|---|
NVFP4-Q4_K_M | 6.52 GiB | 9914 | 147.5 |
NVFP4-Q8_0 | 7.83 GiB | 9587 | 129.3 |
Usage
llama-cli -m Gemma-4-12B-IT-NVFP4-Q4_K_M.gguf -ngl 999 -c 8192 -p "Your prompt here"
llama-server -m Gemma-4-12B-IT-NVFP4-Q4_K_M.gguf -ngl 999 -c 32768 --host 0.0.0.0 --port 8080
Gemma 4 is a reasoning model — it emits a thinking trace before its answer by default.
About the architecture
Gemma-4-12B-IT is an ~11.9B-parameter dense decoder (48 layers) with a hybrid local-sliding + global attention scheme — every 6th layer is a full-attention layer with a larger unified-KV head dim, the rest are sliding-window. The NVFP4 quantization covers the 48 × 3 dense FFN projections; attention projections, embeddings and lm_head stay at higher precision in the source (that's why the -BF16 variant is comparatively large).
Sources & credits
- Base model: google/gemma-4-12B-it by Google — Gemma license
- NVFP4 calibration: self-quantized with NVIDIA ModelOpt 0.44 on an RTX 5090 → LibertAIDAI/Gemma-4-12B-IT-NVFP4
- Tooling: llama.cpp
convert_hf_to_gguf.pyandllama-quantize
License
Inherits the Gemma license from the upstream model.