gemma-4-26B-A4B-it-ExpertsINT4-FP8

Model Overview

This is a mixed-precision quantization of google/gemma-4-26B-A4B-it optimized for inference. The 128 MoE expert weights per layer are quantized to INT4 with GPTQ (group size 64); non-expert transformer linear layers use FP8 dynamic quantization.

• Model Architecture: Gemma4ForConditionalGeneration
  • Input: Text / Image
  • Output: Text
• Model Optimizations:
  • Weight quantization: INT4 (GPTQ, group size 64) for MoE expert weights; FP8 for all other linear layers
  • Activation quantization: FP8 (dynamic, per-token) for non-expert linear layers; none for experts (W4A16)
  • Unquantized: router, embeddings, lm_head, vision tower
• Intended Use Cases: same as the base model — multimodal chat, instruction following, reasoning, tool use
• Release Date: 2026-09-03
• Version: 1.0
• Base model developers: Google DeepMind
• Quantization and release: Systalyze

The base model is a 30-layer / 128-expert MoE. Expert weights dominate the parameter count, but each token touches only a few of them, so experts are stored in INT4 while the always-resident path (attention, dense MLP, router, embeddings, vision tower) stays at FP8 or higher. The checkpoint is stored in the compressed-tensors format and loads directly in vLLM.

3.1× smaller than BF16, and small enough to serve from a single 24 GB card's worth of weights (KV cache and activations are extra).

Models compared

Every number below was measured here, on the same vLLM stack, against these checkpoints of the same base model. The BF16 base appears in the accuracy tables only, the AWQ checkpoint in the throughput tables only:

Throughput

Measured with aiperf against vLLM on 8×H200, ISL ≈ 6,883 / OSL ≈ 322, completion_frac = 1.0 on every cell. The three checkpoints marked above use the same harness, same box, same request stream.

out_tok_per_gpu_s — higher is better:

Latency at the same points (p50 TTFT / p50 ITL, ms):

The win is concurrency-dependent. Mixed precision leads in the memory-bound regime (concurrency 1–8, peaking at +54% over FP8 at concurrency 8) and trails FP8 once the workload becomes compute-bound (concurrency 16–32, down to −15%). If you serve at high concurrency, plain FP8 is the better choice. ExpertsINT4 beats the all-INT4 W4A16 checkpoint at every concurrency point.

Accuracy

All three checkpoints were evaluated in the same session, on the same stack and the same hardware, over full test sets (no subsampling). Higher is better.

Setup: lm_eval 0.4.13 against vLLM 0.27.1 (OpenAI-compatible API), one H200 per checkpoint, identical chat template for all three arms, --apply_chat_template, greedy decoding (temperature=0), --seed 1234, enable_thinking=False. Thinking-mode results are reported separately below.

Sanity check on the harness: our measured base scores land within about 1.4 points of the figures Red Hat AI publishes for this same base model (IFEval 89.28 here, MMLU-Pro 82.06, GPQA Diamond 72.22), so the numbers above are comparable to the wider literature even though every value in the table was measured here.

MMLU-ProX lite, per language (5-shot)

Context retrieval robustness (RULER, 0-shot, 4K / 8K)

Measured separately, ExpertsINT4 against FP8-dynamic only, on 1xH100 with vLLM 0.28.0 and lm_eval 0.4.12, limit=200 per cell.

Results move in both directions relative to FP8, with no consistent degradation across the evaluated 4K and 8K cells. Per-cell standard error is roughly +-0.035. These results evaluate only 4K and 8K contexts and do not evaluate the model's full supported context window.

Thinking mode

Gemma 4 can also be served with thinking enabled. Measured that way the three checkpoints agree to within about a point of each other, and accuracy is clearly higher than with thinking off. One behaviour drives everything else: on some questions the model never closes its thought channel, exhausts the token budget and returns no answer, which scores zero. Raising the cap from 16k to 32k tokens does not remove it and the unquantized BF16 base is affected as much as the quantized checkpoints, so each cell gives accuracy over the questions that produced an answer, with the share that produced none beside it. Budgets were 8k-30k output tokens per task.

Scope

Except where the thinking-mode section says otherwise, every number on this card was produced with thinking disabled, which is the mode we would recommend for throughput-sensitive serving. The throughput and RULER measurements are thinking-disabled only.

Usage

from vllm import LLM, SamplingParams

llm = LLM(model="systalyze/gemma-4-26B-A4B-it-ExpertsINT4-FP8", max_model_len=12288)
out = llm.generate(
    ["Explain mixture-of-experts routing in two sentences."],
    SamplingParams(temperature=1.0, top_p=0.95, top_k=64, max_tokens=256),
)
print(out[0].outputs[0].text)

Sampling defaults follow the base model's generation_config.json (temperature=1.0, top_p=0.95, top_k=64).

License

Apache 2.0, inherited from google/gemma-4-26B-A4B-it.