Gemma 4 12B Instruct - NVFP4 GGUF
Base Model
This is a quantized version of google/gemma-4-12B-it, Google's instruction-tuned 12 billion parameter language model from the Gemma 4 family.
Gemma 4 12B is a dense, decoder-only transformer trained on a diverse dataset of text and code. It excels at instruction following, reasoning, coding, and multimodal understanding.
About This Quantization
- Format: NVFP4 (NVIDIA FP4) — 4-bit E2M1 values with E4M3 scaling per 16 values
- BPW: 4.68 bits per weight
- File size: 6.5 GB
- Hardware target: NVIDIA Blackwell GPUs (RTX 50 series, SM 120a)
- Note: NVFP4 decode is kernel-limited on consumer GPUs (~50 t/s on RTX 5060 Ti). The FP4 tensor cores primarily accelerate prefill, not decode. MXFP4 variant available for faster decode.
Quantization Details
| Property | Value |
|---|---|
| Base model | google/gemma-4-12B-it |
| Architecture | Gemma4ForCausalLM |
| Parameters | 12.08B |
| Layers | 48 |
| Hidden size | 3840 |
| Context length | 128K (tested up to 1M with Q4_0 KV) |
| Quantization | NVFP4 |
| BPW | 4.68 |
| File size | 6.5 GB |
Performance (RTX 5060 Ti 16GB)
| Context | KV Cache | Prompt | Generation |
|---|---|---|---|
| 128K | Q8_0 | 43.8 t/s | 49.9 t/s |
| 256K | Q8_0 | 244.8 t/s | 49.5 t/s |
| 256K | Q4_0 | 309.8 t/s | 50.8 t/s |
| 512K | Q4_0 | 301.0 t/s | 50.0 t/s |
| 1M | Q4_0 | 47.4 t/s | 50.2 t/s |
| 1M | Q8_0 | 17.4 t/s | 27.8 t/s |
Recommended Settings
llama-cli -m gemma-4-12b-it-nvfp4.gguf -ngl 99 -c 262144 -ctk q8_0 -ctv q8_0 -b 1024 -ub 512 --flash-attn
-ngl 99— full GPU offload-c 262144— 256K context (recommended sweet spot)-ctk q8_0 -ctv q8_0— KV cache at Q8_0 for best quality-b 1024 -ub 512— optimized batch sizes--flash-attn— essential for long context
Why No MTP?
Gemma 4 models do not include Multi-Token Prediction (MTP) heads. MTP is a Qwen3-specific architecture feature. This is a standard Gemma 4 model with no modifications to the architecture.
Usage with llama.cpp
# Interactive mode
./llama-cli -m gemma-4-12b-it-nvfp4.gguf -ngl 99 -c 262144 -ctk q8_0 -ctv q8_0 --flash-attn -i
# Server mode
./llama-server -m gemma-4-12b-it-nvfp4.gguf -ngl 99 -c 262144 -ctk q8_0 -ctv q8_0 --flash-attn --host 0.0.0.0 --port 8080
License
Apache 2.0 — same as the base model.