Description
Gemma 4 26B A4B NVFP4 GGUF created from https://huggingface.co/nvidia/Gemma-4-26B-A4B-NVFP4
Model Overview
Gemma 4 26B IT is an open multimodal model built by Google DeepMind that handles text and image inputs, can process video as sequences of frames, and generates text output. It is designed to deliver frontier-level performance for reasoning, agentic workflows, coding, and multimodal understanding on consumer GPUs and workstations, with a 256K-token context window and support for over 140 languages. The model uses a hybrid attention mechanism that interleaves local sliding-window and full global attention, with unified Keys and Values in global layers and Proportional RoPE (p-RoPE) to support long-context performance.
This repository contains a GGUF conversion of the NVIDIA NVFP4-quantized checkpoint, making it usable with llama.cpp and other GGUF-compatible inference engines.
Model Architecture
| Property | Value |
|---|---|
| Architecture Type | Transformers |
| Total Parameters | 25.2B |
| Active Parameters | 3.8B |
| Layers | 30 |
| Sliding Window | 1024 tokens |
| Context Length | 256K tokens |
| Vocabulary Size | 262K |
| MoE Configuration | 8 active / 128 total experts + 1 shared |
| Supported Modalities | Text, Image |
| Vision Encoder Parameters | ~550M |
GGUF File
| File | Description |
|---|---|
gemma-4-26b-a4b-nvfp4.gguf | Single-file GGUF model (NVFP4 quantization) |
A chat_template.jinja file is also provided for use with chat-based inference.
Usage
llama.cpp
# Build llama.cpp with CUDA support (recommended for NVFP4)
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release
# Run inference
./build/bin/llama-cli \
-m gemma-4-26b-a4b-nvfp4.gguf \
-p "Explain quantum computing in simple terms." \
--temp 1.0 --top-k 64 --top-p 0.95
llama-server (OpenAI-compatible API)
./build/bin/llama-server \
-m gemma-4-26b-a4b-nvfp4.gguf \
--host 0.0.0.0 --port 8080
Generation Parameters
The recommended generation parameters (from the original model's generation_config.json):
| Parameter | Value |
|---|---|
| Temperature | 1.0 |
| Top-K | 64 |
| Top-P | 0.95 |
| BOS Token ID | 2 |
| EOS Token IDs | 1, 106, 50 |
| Pad Token ID | 0 |
Evaluation Results (copied from: nvidia/Gemma-4-26B-A4B-NVFP4)
| Benchmark | Baseline (Full Precision) | NVFP4 |
|---|---|---|
| GPQA Diamond | 80.30% | 79.90% |
| AIME 2025 | 88.95% | 90.00% |
| MMLU Pro | 85.00% | 84.80% |
| LiveCodeBench (pass@1) | 80.50% | 79.80% |
| IFBench | 77.77% | 78.10% |
| IFEval | 96.60% | 96.40% |
Baseline: Gemma-4-26B-A4B-it-bf16 Benchmarked with temperature=1.0, top_p=0.95, max_new_tokens=131072
Acknowledgements
- Original model: google/gemma-4-26B-A4B-it
- NVFP4 quantization: nvidia/Gemma-4-26B-A4B-NVFP4 (via NVIDIA Model Optimizer)