Gemma 4 26B A4B IT QAT — NVFP4 GGUF

NVFP4 GGUF conversions derived from the Quantization-Aware Training (QAT) checkpoint of Gemma 4 26B A4B IT (google/gemma-4-26B-A4B-it-qat-q4_0-unquantized).

This repository provides multiple GGUF quantization levels, making the model usable with llama.cpp and other GGUF-compatible inference engines.

Model Overview

Gemma 4 26B A4B IT is a multimodal model built by Google DeepMind that handles text, image, and audio inputs and generates text output. It is designed for efficient on-device and server deployment.

Model Architecture

GGUF Files

A chat_template.jinja file is also provided for use with chat-based inference.

Usage

llama.cpp (CLI)

# Run text-only inference
./llama-cli \
  -m gemma-4-26b-a4b-qat-nvfp4.gguf \
  -p "Explain quantum computing in simple terms." \
  --temp 1.0 --top-k 64 --top-p 0.95

llama-server (OpenAI-compatible API)

# Text-only
./llama-server \
  -m gemma-4-26b-a4b-qat-nvfp4.gguf \
  --host 0.0.0.0 --port 8080

# Multimodal (image + audio)
./llama-server \
  -m gemma-4-26b-a4b-qat-nvfp4.gguf \
  --mmproj mmproj.gguf \
  --host 0.0.0.0 --port 8080

lllama.cpp with MTP draft

# fit must be off to eanble MTP
# i find max 6, min 1 to be optimal settings
./llama-cli \
  -m gemma-4-26b-a4b-qat-nvfp4.gguf \
  --fit off \ 
  --spec-type draft-mtp -md mtp-nvfp4.gguf \
  --spec-draft-n-max 6 --spec-draft-n-min 1

Multimodal (Image / Audio)

For image and audio inputs, use llama-server or llama-cli with the --mmproj flag pointing to mmproj.gguf. Refer to your inference engine's documentation for passing image/audio data alongside text prompts.

Modality order tip: For best results, place image content before text and audio content after text in your prompt.

Generation Parameters

Recommended parameters:

Thinking Mode

Gemma 4 supports configurable thinking (reasoning) mode:

• Enable: Include the <|think|> token at the start of the system prompt.
• Output format: When thinking is enabled, the model outputs internal reasoning followed by the final answer:

  <|channel>thought
  [Internal reasoning]
  <channel|>
  [Final answer]
  

• Disable: Omit the <|think|> token.

Key Features

• Multimodal: Text, image, and audio understanding
• Long Context: 256K token context window
• Function Calling: Native support for structured tool use (agentic workflows)
• Multilingual: Support for 140+ languages
• Native System Prompt: Supports the system role for structured conversations

Acknowledgements

• Original model: google/gemma-4-26B-A4B-it
• QAT checkpoint: google/gemma-4-26B-A4B-it-qat-q4_0-unquantized
• Technical report: Gemma 4 Technical Report (arXiv:2607.02770)

Citation

@misc{gemmateam2026gemma4,
      title={Gemma 4 Technical Report},
      author={Gemma Team},
      year={2026},
      eprint={2607.02770},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2607.02770},
}

License

Apache License 2.0