Qwen3-VL-8B-Instruct-W4A16-AutoRound-AWQ
Model Overview
This model is the AWQ (Activation-aware Weight Quantization) export of Qwen/Qwen3-VL-8B-Instruct.
It combines the speed of AWQ with the accuracy of AutoRound. The weights were fine-tuned for 800 steps to ensure the 4-bit degradation is negligible. The vision tower remains in full precision (FP16) to maintain top-tier performance on OCR and visual tasks.
Key Features
- High Performance: Optimized for
vLLMserving andTransformers. - Low VRAM: Fits comfortably on 12GB+ GPUs (RTX 3060/4070).
- Accurate Vision: Vision encoder is NOT quantized, preserving full visual acuity.
Usage
Option A: vLLM (Recommended for Speed)
pip install vllm
from vllm import LLM, SamplingParams
model_id = "Vishva007/Qwen3-VL-8B-Instruct-W4A16-AutoRound-AWQ"
llm = LLM(
model=model_id,
quantization="awq",
trust_remote_code=True,
max_model_len=4096
)
# ... (Standard vLLM inference code)
Option B: Transformers (Standard)
pip install autoawq transformers
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
import torch
model_id = "Vishva007/Qwen3-VL-8B-Instruct-W4A16-AutoRound-AWQ"
# Load with Flash Attention 2 for best performance
model = Qwen2VLForConditionalGeneration.from_pretrained(
model_id,
torch_dtype=torch.float16,
device_map="auto",
attn_implementation="flash_attention_2"
)
processor = AutoProcessor.from_pretrained(model_id)
# Inference Example
messages = [
{
"role": "user",
"content": [
{"type": "image", "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"},
{"type": "text", "text": "What does this image show?"},
],
}
]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
text=[text],
images=image_inputs,
videos=video_inputs,
padding=True,
return_tensors="pt",
).to("cuda")
generated_ids = model.generate(**inputs, max_new_tokens=128)
print(processor.batch_decode(generated_ids, skip_special_tokens=True))
🚀 Deploy on RunPod
One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.
🎁 Need GPU compute? Sign up via RunPod and get $5–$500 in free credits when you add your first $10.
PyTorch 2.14
| Template | CUDA Version | Docker Image | Template ID | Deploy |
|---|---|---|---|---|
| PyTorch 2.14 (CUDA 12.6) | 12.6 | vishva123/cuda-12.6-pytorch-2.14-runpod | d7lxsa4w9m |  |
| PyTorch 2.14 (CUDA 13.0) | 13.0 | vishva123/cuda-13.0-pytorch-2.14-runpod | yk0y6j6rpg | |
| PyTorch 2.14 (CUDA 13.2) | 13.2 | vishva123/cuda-13.2-pytorch-2.14-runpod | gsp4gwx0nw | |
PyTorch 2.13
| Template | CUDA Version | Docker Image | Template ID | Deploy |
|---|---|---|---|---|
| PyTorch 2.13 (CUDA 12.6) | 12.6 | vishva123/cuda-12.6-pytorch-2.13-runpod | gmlupxnxfk | |
| PyTorch 2.13 (CUDA 13.0) | 13.0 | vishva123/cuda-13.0-pytorch-2.13-runpod | y3j8xvk4f4 | |
| PyTorch 2.13 (CUDA 13.2) | 13.2 | vishva123/cuda-13.2-pytorch-2.13-runpod | vigpissn5w | |
PyTorch 2.12
| Template | CUDA Version | Docker Image | Template ID | Deploy |
|---|---|---|---|---|
| PyTorch 2.12 (CUDA 12.6) | 12.6 | vishva123/cuda-12.6-pytorch-2.12-runpod | ctmz86zmf0 | |
| PyTorch 2.12 (CUDA 13.0) | 13.0 | vishva123/cuda-13.0-pytorch-2.12-runpod | qjko5yiwzi | |
| PyTorch 2.12 (CUDA 13.2) | 13.2 | vishva123/cuda-13.2-pytorch-2.12-runpod | ifg6xmye0f | |
Citation
@misc{qwen3technicalreport,
title={Qwen3 Technical Report},
author={Qwen Team},
year={2025}
}