Gemma 4 26B A4B IT — MXFP4-MOE GGUF

Model Description

This is an MXFP4_MOE-quantized GGUF version of google/gemma-4-26B-A4B-it, Google's state-of-the-art 26B-parameter MoE instruction-tuned model (4B active parameters, 128 experts, 8 experts per token).

MXFP4_MOE is a specialized MXFP4 quantization variant designed for Mixture-of-Experts architectures, providing efficient 4-bit quantization while preserving the expert routing quality.

This repository includes the multimodal vision encoder (mmproj) for image understanding capabilities.

Quantization Details

Files

Usage

llama.cpp

# Text-only inference
./llama-cli -m gemma-4-26b-MXFP4-MOE.gguf -p "Hello, how are you?" -n 256

# Multimodal (image) inference
./llama-cli -m gemma-4-26b-MXFP4-MOE.gguf --mmproj mmproj-gemma-4-26b-f16.gguf \
  --image path/to/image.jpg -p "Describe this image" -n 256

Inference with llama.cpp servers

./llama-server -m gemma-4-26b-MXFP4-MOE.gguf --mmproj mmproj-gemma-4-26b-f16.gguf \
  --host 0.0.0.0 --port 8080

Technical Notes

• Thinking is disabled by default. To enable thinking, pass enable_thinking=true in your request.
• Gemma-4 uses a hybrid attention mechanism with sliding window (local) and global (full) attention layers in a 5:1 pattern.
• The model uses proportional RoPE for global attention layers and standard RoPE for sliding window layers.
• MXFP4_MOE is designed specifically for MoE architectures and provides better quality than general-purpose quantization formats for expert-heavy models.
• CPU and GPU inference are both supported.

Hardware Requirements

• RAM: 8 GB+ (model fits in most modern GPUs at 4-bit)
• Storage: ~15 GB for the model files

Acknowledgements

• Google for the original Gemma 4 model
• llama.cpp for the GGUF format and quantization tools
• AMD for the MXFP4 specification