granite-4.1-3b-quantized.w8a8

INT8 (W8A8) compressed-tensors quantization of ibm-granite/granite-4.1-3b.

• Weights: INT8, symmetric, per-channel
• Activations: INT8, symmetric, dynamic per-token
• Scope: only Linear layers inside the transformer blocks; lm_head is left in BF16 (the base model has tie_word_embeddings: true, so quantizing it would also perturb the input embedding)
• Method: post-training, one-shot SmoothQuant → GPTQ via llm-compressor
• Size: 3.89 GiB on disk. The linear weights halve; the tied embedding matrix, the norms and lm_head stay BF16, so the whole-checkpoint saving is smaller than 2× (and smaller the smaller the model, since the 100k-entry vocab is a larger share of it)
• Tooling: llmcompressor 0.9.0.4, compressed-tensors 0.13.0, transformers 4.57.3

Purpose. This checkpoint was produced for inference-performance benchmarking (INT8/AMX on Xeon and INT8 kernels on GPU). No accuracy evaluation was run on it — see Accuracy before using it for anything where quality matters.

Deployment with vLLM

vllm serve devpramod-intel/granite-4.1-3b-quantized.w8a8 --max-model-len 32768

from vllm import LLM, SamplingParams
from transformers import AutoTokenizer

model_id = "devpramod-intel/granite-4.1-3b-quantized.w8a8"
tokenizer = AutoTokenizer.from_pretrained(model_id)
llm = LLM(model=model_id, max_model_len=4096)

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Who are you? Please respond in pirate speak!"}],
    tokenize=False, add_generation_prompt=True,
)
print(llm.generate(prompt, SamplingParams(temperature=0.3, max_tokens=256))[0].outputs[0].text)

Creation

python quantize_w8a8_granite41.py \
  --model-dir ibm-granite/granite-4.1-3b \
  --out granite-4.1-3b-quantized.w8a8 \
  --smoothing-strength 0.8 --dampening-frac 0.1 \
  --observer mse --num-samples 512

Recipe:

quant_stage:
  quant_modifiers:
    SmoothQuantModifier:
      smoothing_strength: 0.8
      ignore: [lm_head]
      mappings:
      - - ['re:.*q_proj', 're:.*k_proj', 're:.*v_proj']
        - re:.*input_layernorm
      - - ['re:.*gate_proj', 're:.*up_proj']
        - re:.*post_attention_layernorm
      - - ['re:.*down_proj']
        - re:.*up_proj
    GPTQModifier:
      targets: [Linear]
      ignore: [lm_head]
      scheme: W8A8
      dampening_frac: 0.1
      weight_observer: mse
      sequential_targets: [GraniteDecoderLayer]

recipe.yaml in this repo is what llm-compressor actually applied and is authoritative. It additionally shows block_size: 128 and actorder: static, which are llm-compressor 0.9.0.4 defaults rather than choices — the older Granite cards predate actorder defaulting on, so this checkpoint is not bit-identical to what their recipe produced in 2025.

Calibration: neuralmagic/LLM_compression_calibration, train split, shuffle(seed=42).select(512), the dataset's raw text field with add_special_tokens=True, max_seq_length=8192.

Recipe provenance

Every knob is taken from Red Hat AI's published recipe.yaml files for the nearest architectural precedents — ibm-granite/granite-4.1-3b is a dense GraniteForCausalLM with Llama-style blocks (q/k/v + gate/up/down, RMSNorm), so the Granite 3.1 W8A8 recipes transfer directly.

Deliberate deviations from those cards:

• 512 calibration samples instead of the Granite cards' 3072 — W8A8 is far less calibration-sensitive than W4A16, and 512 is the llm-compressor default.
• max_seq_length=8192, not the 8196 printed on the Granite cards (a typo).
• sequential_targets set to the decoder-layer class, following current Red Hat cards; it lowers peak VRAM and does not change the result.

Accuracy

No accuracy benchmark was run on this checkpoint. It exists to measure throughput and latency. The figures below are estimates by precedent, not measurements of this model, and should not be quoted as such:

On that basis the expected recovery here is ~99–100% on knowledge/reasoning multiple-choice suites and ~98–99% on generative suites. If you need a number you can defend, run lm-eval against both this checkpoint and the BF16 base and report the ratio.

Verification performed

• config.json → quantization_config: format: int-quantized, weights num_bits 8 / channel / symmetric / observer mse, input activations num_bits 8 / token / dynamic, ignore: ["lm_head"]
• all quantized weights and scales checked finite (no NaN/Inf)
• checkpoint loads and generates coherent text