Gemma 4 E4B Instruct — W4A16 Quantized GPTQ

This repository hosts W4A16 INT4-quantized versions of google/gemma-4-E4B-it, a multimodal mixture-of-experts model supporting text, vision, and audio inputs. Two quantized variants are available:

Note on MTP / Speculative Decoding:
If you want to use the official speculative decoding assistant model (google/gemma-4-E4B-it-assistant) for MTP support, it is recommended to use the vllm/vllm-openai:gemma4-0505-cu129 Docker image, which includes newer Gemma 4 support and decoding patches.

Due to INT4 quantization, the assistant acceptance rate may be lower compared to the original unquantized google/gemma-4-E4B-it model.

Note: Only the language model (LM) layers are quantized to INT4. The vision tower, audio tower, and multimodal projectors are kept at full precision (BF16) to preserve multimodal quality.

Quantization Details

Model Architecture

Gemma 4 E4B is a multimodal MoE model (Gemma4ForConditionalGeneration) with:

• Text backbone: 42-layer Gemma4TextModel with 2560 hidden dim, mixed local/global attention
• Vision tower: 16-layer Gemma4VisionModel (768-dim, unquantized)
• Audio tower: 12-layer Gemma4AudioModel with conformer-style layers (unquantized)
• Vocabulary size: 262,144 tokens

Usage

vLLM Inference

The recommended way to serve this model is via the official vllm/vllm-openai:gemma4 Docker image, which ships vLLM v0.19.1 with the latest Transformers patches required for Gemma 4.

Serve with Docker (recommended)

docker run --gpus all --rm -p 8000:8000 \
  vllm/vllm-openai:gemma4 \
  vllm serve Vishva007/gemma-4-E4B-it-W4A16-AutoRound-GPTQ \
    --served-model-name Gemma-4-E4B-it \
    --quantization autoround \
    --kv-cache-dtype auto \
    --max-num-batched-tokens 16384 \
    --enable-chunked-prefill \
    --enable-prefix-caching \
    --dtype bfloat16 \
    --max-model-len 18432 \
    --enable-auto-tool-choice \
    --tool-call-parser gemma4 \
    --port 8000 \
    --default-chat-template-kwargs '{"enable_thinking": false}' \
    --mm-processor-kwargs '{"max_soft_tokens": 560}'

Direct vllm serve (vLLM ≥ 0.19.0)

vllm serve Vishva007/gemma-4-E4B-it-W4A16-AutoRound-GPTQ \
  --served-model-name Gemma-4-E4B-it \
  --quantization autoround \
  --kv-cache-dtype auto \
  --max-num-batched-tokens 16384 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --dtype bfloat16 \
  --max-model-len 18432 \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --port 8000 \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --mm-processor-kwargs '{"max_soft_tokens": 560}'

max_soft_tokens — Image Token Budget

The max_soft_tokens parameter controls how many visual tokens are allocated per image. Higher values give richer image representations at the cost of context length and throughput.

Pass it via --mm-processor-kwargs '{"max_soft_tokens": <value>}'.

OpenAI-compatible API call

Once the server is running, query it like any OpenAI-compatible endpoint:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="Gemma-4-E4B-it",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe what you see."},
                {"type": "image_url", "image_url": {"url": "https://..."}},
            ],
        }
    ],
    max_tokens=512,
)
print(response.choices.message.content)

The quantized model was then exported in both AutoRound and GPTQ formats and pushed to Hugging Face Hub.

Limitations & Notes

• RTN mode (iters=0) is used instead of full AutoRound optimization due to Gemma 4's architecture constraints.
• Some layers with shapes not divisible by 32 are skipped during quantization (minor precision impact).
• Multimodal (vision/audio) capabilities are fully preserved as those towers are not quantized.

🚀 Deploy on RunPod

One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.

🎁 Need GPU compute? Sign up via RunPod and get $5–$500 in free credits when you add your first $10.

PyTorch 2.14

Template	CUDA Version	Docker Image	Template ID	Deploy
PyTorch 2.14 (CUDA 12.6)	12.6	vishva123/cuda-12.6-pytorch-2.14-runpod	d7lxsa4w9m
PyTorch 2.14 (CUDA 13.0)	13.0	vishva123/cuda-13.0-pytorch-2.14-runpod	yk0y6j6rpg
PyTorch 2.14 (CUDA 13.2)	13.2	vishva123/cuda-13.2-pytorch-2.14-runpod	gsp4gwx0nw

PyTorch 2.13

Template	CUDA Version	Docker Image	Template ID	Deploy
PyTorch 2.13 (CUDA 12.6)	12.6	vishva123/cuda-12.6-pytorch-2.13-runpod	gmlupxnxfk
PyTorch 2.13 (CUDA 13.0)	13.0	vishva123/cuda-13.0-pytorch-2.13-runpod	y3j8xvk4f4
PyTorch 2.13 (CUDA 13.2)	13.2	vishva123/cuda-13.2-pytorch-2.13-runpod	vigpissn5w

PyTorch 2.12

Template	CUDA Version	Docker Image	Template ID	Deploy
PyTorch 2.12 (CUDA 12.6)	12.6	vishva123/cuda-12.6-pytorch-2.12-runpod	ctmz86zmf0
PyTorch 2.12 (CUDA 13.0)	13.0	vishva123/cuda-13.0-pytorch-2.12-runpod	qjko5yiwzi
PyTorch 2.12 (CUDA 13.2)	13.2	vishva123/cuda-13.2-pytorch-2.12-runpod	ifg6xmye0f

Acknowledgements

Special thanks to OLAF-OSS and the contributors of the gemma4-vllm project for their excellent work in enabling Gemma 4 support with vLLM. Their QUANTIZE.md guide was extremely helpful in understanding the correct quantization approach for gemma-4-E2B-it.

At the time of writing, the original repository appears to be unavailable or removed. To ensure reproducibility, I have documented the full quantization workflow used for this model in my own repository.

The full quantization process used to produce these models is documented here: 📓 auto_round_Gemma4-E4B.ipynb

License

This quantized model is derived from google/gemma-4-E4B-it and is subject to the Gemma Terms of Use.

Citation

If you use this quantized model, please also cite the original Gemma 4 work:

@misc{gemma4_2026,
  title  = {Gemma 4},
  author = {Google DeepMind},
  year   = {2026},
  url    = {https://huggingface.co/google/gemma-4-E4B-it}
}

Quantized by Vishva007