gemma-4-12B-it-Mixed-8bit
Mixed-precision 8-bit quantization of Gemma 4 12B IT — Google's unified multimodal model (text + vision + audio + video), optimized for efficient local inference with minimal quality loss.
Quick Stats
| Metric | Value |
|---|---|
| Base model | google/gemma-4-12b-it |
| Architecture | Gemma4Unified (48-layer dense, hybrid attention) |
| Quantization | MLP-only 8-bit, group-size=64 |
| BF16 size | ~24 GB |
| Mixed 8-bit size | ~15 GB (≈38% reduction) |
| Modalities | Text, Vision, Audio, Video |
| License | Gemma License (inherits from base model) |
Quantization Strategy
This model uses surgical mixed-precision quantization: only the SwiGLU MLP projection matrices (gate_proj, up_proj, down_proj) in the language model are quantized to 8-bit. All other components remain in BF16:
- ✅ Quantized (8-bit): LLM MLP layers (144 tensors across 48 layers, ~60% of model mass)
- ✅ Protected (BF16): All attention (sliding + full), vision tower, audio embedder, embeddings, LM head, MultimodalEmbedder, all normalization layers, layer_scalars
Rationale: MLP layers are "knowledge lookup" / feature transformation layers that are extremely quantization-tolerant. Attention mechanisms carry reasoning and context-window recall. Vision, audio, and multimodal projection layers are kept in BF16 to preserve modality fidelity.
Model Architecture
Language Model (Text)
- 48 dense layers in pattern
[sliding_attention × 5, full_attention × 1] × 8 - Hybrid attention: Sliding window attention (40 layers, head_dim=256) + full global attention (8 layers, head_dim=512)
- Heads: 16 query / 8 KV / 1 global KV
- Hidden size: 3840, Intermediate: 15360
- Vocab: 262,144, Context: 262,144 tokens
- RoPE: Default (θ=10,000) for sliding; Proportional (θ=1,000,000, partial_rotary=0.25) for full attention
- No MoE (dense architecture)
Vision Tower
- ViT with patch_size=16, pooling_kernel=3
- 280 soft tokens per image, projected to 3840-dim
- Position embedding size: 1120
Audio Tower
- Audio embedder: 640-dim embedding, 640 samples per token
- Simple linear projection (no convolutional audio tower in this checkpoint)
Multimodal
- Unified processor:
Gemma4UnifiedProcessorhandling text, images, audio, and video - Special tokens:
<|image|>,<|audio|>,<|video|>with BOI/EOI, BOA/EOA delimiters - MultimodalEmbedder projects vision and audio into the shared 3840-dim text space
MTP Speculative Decoding
This model supports Multi-Token Prediction (MTP) speculative decoding via the companion assistant/drafter model. This model achieves 20-30% faster token generation with gemma-4-12B-it-assistant in BF16 format.
Usage
oMLX (Recommended)
Loads natively with MTP and vision support.
MLX / mlx-vlm
from mlx_vlm.utils import load
model, processor = load("your-username/gemma-4-12B-it-Mixed-8bit", lazy=True)
# Model loads in ~15 GB vs ~24 GB for BF16
Files
| File | Description |
|---|---|
model-00001.safetensors – model-00003.safetensors | Sharded model weights (~5 GB each) |
config.json | Model configuration with quantization metadata |
tokenizer.json / tokenizer_config.json | Tokenizer (262K vocab) |
chat_template.jinja | Chat template with thinking/tool support |
generation_config.json | Generation parameters |
processor_config.json | Unified processor config |
Quantization Details
| Parameter | Value |
|---|---|
| Quantization mode | Affine (8-bit) |
| Group size | 64 |
| Layers quantized | 48 (all MLP: gate_proj, up_proj, down_proj) |
| Tensors quantized | ~144 |
| Layers protected | Attention (Q/K/V/O), Vision, Audio, Embeddings, Norms, Scalars |
| Script | quantization_scripts/mixed_quantize_gemma4_12b.py |
Reproducing
# Quantize from the BF16 base model
mlx-env/bin/python quantization_scripts/mixed_quantize_gemma4_12b.py \
--bits 8 --group-size 64
The script:
- Lazily loads the BF16 checkpoint via
mlx_vlm.utils.load() - Applies 8-bit quantization to MLP projections only (via
class_predicatefilter) - Saves config with quantization metadata
- Flattens, evaluates, and shards to 5 GB safetensors
Limitations
- Mixed quantization may show minor quality degradation on edge cases compared to BF16
- The vision tower, audio embedder, and multimodal projector are kept in BF16 — they are not quantized in this version
- Audio support uses a simple linear projection (no convolutional audio tower); complex audio understanding may be limited
- MTP speculative decoding requires the companion assistant model to be loaded alongside
- The base model uses
transformers_version: 5.10.0.dev0— some fields may not be recognized by stable transformers releases yet
License
This model inherits the Apache 2.0 license of the base google/gemma-4-12b-it.