Qwen3-VL-8B-Instruct-w8a8-llmcompressor-v0.12.0

Model Overview

• Model Architecture: Qwen3VLForConditionalGeneration
  • Input: Text, Image
  • Output: Text
• Source Model: Qwen3-VL-8B-Instruct
• Supported Hardware: AMD EPYC (CPU inference)
• Preferred Operating System: Linux
• Inference Engine: vLLM v0.26.0
• Quantization Framework: LLM Compressor v0.12.0
• Quantization Method: 8-bit Weight, 8-bit Dynamic Activation Quantization (W8A8)
• Compatible Stack:
  • ZenDNN v6.1.0
  • ZenTorch v2.11.0.3
  • PyTorch v2.11.0
  • LLM Compressor v0.12.0
  • vLLM v0.26.0

This is a quantized version of Qwen3-VL-8B-Instruct created by AMD using LLM Compressor (compressed-tensors) for ZenDNN-optimized CPU inference.

Quantization

The model was quantized from Qwen3-VL-8B-Instruct using LLM Compressor via the Round-to-Nearest (RTN) algorithm. This reduces the model weights from 16.3 GiB to 9.9 GiB on disk (~40% reduction).

• Method: 8-bit Weight, 8-bit Dynamic Activation Quantization (W8A8)
• Config: compressed-tensors, num_bits=8, type=int, symmetric=true
• Weights: INT8, symmetric, per-channel (static)
• Activations: INT8, symmetric, per-token (dynamic)
• Kept in BF16: the full vision tower (model.visual.*, including the attention/MLP blocks, the patch merger, and the deepstack mergers) and lm_head

Only the language-model Linear layers are quantized. The vision encoder and the vision-to-text projector stay in BF16, since calibration statistics come from text and the vision path is comparatively sensitive. That is also why the on-disk reduction here (~40%) is smaller than the ~46-48% typical of text-only W8A8 models.

import torch
import transformers
from transformers import AutoConfig, AutoProcessor, AutoTokenizer

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

model_id = "Qwen/Qwen3-VL-8B-Instruct"
output_dir = "./Qwen3-VL-8B-Instruct-w8a8-llmcompressor-v0.12.0"

# Step 1: Load the BF16 model and tokenizer via the architecture named in the
# model's own config. AutoModelForCausalLM would route to the inner text model
# and save a config.json demoted to text-only, which vLLM rejects.
config = AutoConfig.from_pretrained(model_id, trust_remote_code=True)
model_cls = getattr(transformers, config.architectures[0])
model = model_cls.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="cpu",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

# Step 2: Define the W8A8 recipe. Only the language-model Linears are quantized:
# the vision tower (model.visual.*) and lm_head must stay BF16.
recipe = QuantizationModifier(
    scheme="W8A8",
    targets=["Linear"],
    ignore=[
        "re:.*lm_head",
        "re:.*visual.*",
    ],
)

# Step 3: One-shot quantize and save in compressed-tensors format
oneshot(
    model=model,
    recipe=recipe,
    tokenizer=tokenizer,
    output_dir=output_dir,
    trust_remote_code_model=True,
)

# Step 4: Save the processor; oneshot does not write it and vLLM needs it
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
processor.save_pretrained(output_dir)

# Step 5: Text-only smoke test. VLMs have custom generate signatures, so a
# failure here is not fatal; the checkpoint is already saved above.
try:
    inputs = tokenizer("What are we having for dinner?", return_tensors="pt")
    output = model.generate(**inputs, max_new_tokens=30)
    print(tokenizer.decode(output[0], skip_special_tokens=True))
except Exception as e:
    print(f"Smoke test skipped ({type(e).__name__}: {e})")

Quick Start

Use with vLLM

from vllm import LLM, SamplingParams

model = LLM(
    model="amd/Qwen3-VL-8B-Instruct-w8a8-llmcompressor-v0.12.0",
    dtype="bfloat16",
)

sampling_params = SamplingParams(temperature=0.7, max_tokens=256)
outputs = model.generate(["Hello, how are you?"], sampling_params)
print(outputs[0].outputs[0].text)

Requirements

torch==2.11.0
zentorch==2.11.0.3
vllm==0.26.0
llmcompressor==0.12.0

OpenMP Setup

For optimal performance, set LD_PRELOAD with libomp.so (LLVM OpenMP) or libiomp5.so (Intel OpenMP):

# Using LLVM OpenMP (llvmopenmp)
export LD_PRELOAD=$(find /path/to/env -name "libomp.so" | head -1)

# Or using Intel OpenMP (libiomp)
export LD_PRELOAD=$(find /path/to/env -name "libiomp5.so" | head -1)

Note: Set LD_PRELOAD before launching vLLM or any inference script.

Evaluation

The model was evaluated against the BF16 (unquantized) baseline on multimodal benchmarks using lm-evaluation-harness with the vLLM vision-language engine.

Evaluation Command

lm_eval \
    --model vllm-vlm \
    --model_args pretrained=amd/Qwen3-VL-8B-Instruct-w8a8-llmcompressor-v0.12.0,dtype=bfloat16 \
    --tasks chartqa,mmmu_val_tech_and_engineering \
    --batch_size auto \
    --trust_remote_code \
    --apply_chat_template \
    --log_samples \
    --output_path .

Limitations

• Version Lock: This model is compatible with ZenDNN v6.1.0 / ZenTorch v2.11.0.3 / PyTorch v2.11.0. It may not load correctly on other versions.
• CPU Only: This model is optimized for AMD EPYC CPU inference via ZenDNN. It is not intended for GPU inference.
• Vision Path Unquantized: The vision tower remains in BF16, so the memory saving is smaller than for text-only W8A8 models and image preprocessing cost is unchanged.

License

This model is distributed under the same license as the source model. See the LICENSE file for details.

Modifications copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.