gemma-4-E2B-it-qat-GGUF

gemma-4-E2B-it-qat-q4_0-unquantized is a compact, on-device-optimized instruction-tuned multimodal model from Google DeepMind, part of the Gemma 4 family, featuring 2.3 billion effective parameters (5.1B total including embeddings) optimized via Quantization-Aware Training (QAT) to maintain near-bfloat16 quality at significantly reduced memory footprint. Designed specifically for efficient local execution on laptops and mobile devices, it employs Per-Layer Embeddings (PLE) for maximum parameter efficiency and supports text, image, and audio modalities with a 128K token context window and a 262K vocabulary across 140+ languages. The model uses a hybrid attention mechanism interleaving local sliding window attention (512 tokens) with full global attention across 35 layers, alongside a ~150M parameter vision encoder and ~300M parameter audio encoder, enabling capabilities like ASR, speech translation, image understanding, OCR, video frame analysis, native function calling, and configurable thinking/reasoning mode. The Q4_0 unquantized variant provides half-precision weights extracted from the QAT pipeline, making it suited for custom downstream compilation and research, with benchmark scores of 60.0% on MMLU Pro, 43.4% on GPQA Diamond, 33.47 on CoVoST audio translation, and 19.1% on the 128K long-context MRCR v2 task.

[!note] Google DeepMind’s Gemma 4 Quantization-Aware Training (QAT) releases compress models by simulating lower precision during the training process itself. This drastically reduces VRAM requirements and accelerates local inference on consumer hardware and mobile devices while preserving the near-original quality of uncompressed baselines.

Model Files

llama.cpp

LLM inference in C/C++ — https://github.com/ggml-org/llama.cpp