Gemma-4-31B-it-Quark-W8A8-INT8

W8A8 INT8 quantized version of google/gemma-4-31B-it using AMD Quark.

Model Details

Quantization Scheme

Accuracy

GSM8K 8-shot evaluation on the full 1319-question test split (vLLM, temperature=0, concurrency=16, max_tokens=512, standard chat template with #### answer format):

Δ vs BF16: −0.08pp (essentially lossless).

How to Use

With vLLM (Recommended)

# Start the server (single MI300X / MI350X / MI355X is enough; A100-80G also works)
vllm serve nameistoken/Gemma-4-31B-it-Quark-W8A8-INT8 \
    --tensor-parallel-size 1 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.9 \
    --trust-remote-code

# Chat completion
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "nameistoken/Gemma-4-31B-it-Quark-W8A8-INT8",
  "messages": [{"role": "user", "content": "Hello! What is the capital of France?"}],
  "max_tokens": 256,
  "temperature": 0.7
}'

Hardware Requirements

• Minimum: 1× GPU with ≥48 GB VRAM (e.g., AMD MI300X / MI350X / MI355X, NVIDIA A100-80G / H100).
• For longer context or larger batches use TP=2 across two of the same GPUs.

Quantization Details

This model was quantized using AMD Quark's per-token per-channel INT8 scheme:

• Weight quantization: INT8 per-channel (one scale per output channel), symmetric, static.
• Activation quantization: INT8 per-token (one scale per token), symmetric, dynamic (computed at inference time).
• Excluded layers: lm_head, *embed_tokens*, *vision_tower*, *embed_vision* (output head + token embedding + the entire vision tower remain in BF16).
• Export: pack_method='order', weight_format='real_quantized', custom_mode='quark' → real INT8 weights with BF16 scales (no fake-quant, no zero-point).

Reproduce Quantization

# 1. Environment
pip install amd-quark==0.11.1 datasets accelerate
git clone https://github.com/huggingface/transformers.git
cd transformers && pip install -e . --no-deps   # transformers main (>= 5.6.0.dev0)

# quark_gemma4_int8.py
import os, torch
from transformers import AutoTokenizer, Gemma4ForConditionalGeneration
from quark.torch import ModelQuantizer
from quark.torch.quantization.config.config import (
    QTensorConfig, QuantizationConfig, Config, Dtype,
)
from quark.torch.quantization.config.type import (
    RoundType, ScaleType, QSchemeType,
)
from quark.torch.quantization.observer import PerChannelMinMaxObserver

MODEL_IN  = "google/gemma-4-31B-it"
MODEL_OUT = "./Gemma-4-31B-it-Quark-W8A8-INT8"

tokenizer = AutoTokenizer.from_pretrained(MODEL_IN, trust_remote_code=True)
model = Gemma4ForConditionalGeneration.from_pretrained(
    MODEL_IN, torch_dtype=torch.bfloat16,
    device_map="auto", trust_remote_code=True,
)

weight_spec = QTensorConfig(
    dtype=Dtype.int8, observer_cls=PerChannelMinMaxObserver,
    symmetric=True, is_dynamic=False,
    qscheme=QSchemeType.per_channel, ch_axis=0,
    round_method=RoundType.round, scale_type=ScaleType.float,
)
input_spec = QTensorConfig(
    dtype=Dtype.int8, observer_cls=PerChannelMinMaxObserver,
    symmetric=True, is_dynamic=True,
    qscheme=QSchemeType.per_channel, ch_axis=1,
    round_method=RoundType.round, scale_type=ScaleType.float,
)

q_cfg = Config(
    global_quant_config=QuantizationConfig(
        input_tensors=input_spec, weight=weight_spec,
    ),
    exclude=[
        "lm_head", "*embed_tokens*",
        "*vision_tower*", "*embed_vision*",
    ],
)

quantizer = ModelQuantizer(q_cfg, multi_device=True)
model = quantizer.quantize_model(model, dataloader=None)  # PTQ, no calibration data needed for dynamic act
quantizer.freeze(model)

quantizer.export_model(
    model, MODEL_OUT,
    pack_method="order",
    weight_format="real_quantized",
    custom_mode="quark",
)
tokenizer.save_pretrained(MODEL_OUT)

Citation

If you use this model, please cite the original Gemma 4 release:

@misc{google2026gemma4,
  title  = {Gemma 4},
  author = {Google DeepMind},
  year   = {2026},
  url    = {https://huggingface.co/google/gemma-4-31B-it}
}

License

This model is released under the Apache License 2.0, following the Gemma 4 license under which the upstream google/gemma-4-31B-it weights are distributed by Google DeepMind.

This is a quantized derivative of google/gemma-4-31B-it. Per Apache 2.0 §4:

• Modified files (the INT8-quantized model.safetensors and the appended quantization_config block in config.json) carry this notice as part of the model card.
• Original copyright and attribution notices from the base model are preserved (see NOTICE).
• A copy of the Apache 2.0 license text is included as LICENSE.

Original weights © Google DeepMind. Quantization performed by the model author; no warranty of any kind is provided (see LICENSE §7–8).