Gemma 4 26B A4B IT — NVFP4 GGUF

Model Description

This is an NVFP4-quantized GGUF version of google/gemma-4-26B-A4B-it, Google's state-of-the-art 26B-parameter MoE instruction-tuned model (4B active parameters, 128 experts, 8 experts per token).

NVFP4 is NVIDIA's native Blackwell FP4 quantization format, delivering excellent quality at 4-bit precision with optimized hardware support on Blackwell GPUs.

This repository includes the multimodal vision encoder (mmproj) for image understanding capabilities.

Quantization Details

Files

Usage

llama.cpp

# Text-only inference
./llama-cli -m gemma-4-26b-NVFP4.gguf -p "Hello, how are you?" -n 256

# Multimodal (image) inference
./llama-cli -m gemma-4-26b-NVFP4.gguf --mmproj mmproj-gemma-4-26b-f16.gguf \
  --image path/to/image.jpg -p "Describe this image" -n 256

Inference with llama.cpp servers

./llama-server -m gemma-4-26b-NVFP4.gguf --mmproj mmproj-gemma-4-26b-f16.gguf \
  --host 0.0.0.0 --port 8080

Technical Notes

• Thinking is disabled by default. To enable thinking, pass enable_thinking=true in your request.
• Gemma-4 uses a hybrid attention mechanism with sliding window (local) and global (full) attention layers in a 5:1 pattern.
• The model uses proportional RoPE for global attention layers and standard RoPE for sliding window layers.
• NVFP4 requires llama.cpp built with CUDA support and a Blackwell-compatible GPU for best performance.
• CPU inference is supported but slower than GPU-accelerated inference.

Hardware Requirements

• GPU: Blackwell (RTX 5090, B200, etc.) for optimal NVFP4 performance
• RAM: 8 GB+ (model fits in most modern GPUs at 4-bit)
• Storage: ~14 GB for the model files

Acknowledgements

• Google for the original Gemma 4 model
• llama.cpp for the GGUF format and quantization tools
• NVIDIA for the NVFP4 specification