Qwen3.5-4B-AWQ (W4A16)
This is an AWQ W4A16 (4-bit weight) quantized version of
Qwen/Qwen3.5-4B, packaged in the
compressed-tensors format for efficient inference with
vLLM.
The quantization reduces the on-disk and VRAM footprint while preserving the base model's vision-language capabilities, making it well-suited to a single consumer GPU (e.g. an RTX 5090 / Blackwell, sm_120).
About the base model
Qwen3.5-4B is a compact multimodal vision-language model from the Qwen team (Alibaba Group). It accepts interleaved image + text (and video) input and generates text, with a "thinking" reasoning mode enabled by default.
| Base model | Qwen/Qwen3.5-4B |
| Parameters | ~4B |
| Modality | Image-Text-to-Text (vision-language) |
| Architecture | Hybrid Gated DeltaNet + sparse MoE, with a vision encoder |
| Context length | 262,144 tokens native (extensible with RoPE scaling) |
| Languages | 201 languages and dialects |
| License | Apache 2.0 |
For full details on capabilities, benchmarks, and prompt formatting, see the base model card.
Quantization details
| Method | AWQ (Activation-aware Weight Quantization) |
| Scheme | W4A16 (4-bit weights, 16-bit activations) |
| Format | compressed-tensors (pack-quantized) |
| Quantized modules | Linear layers of the language model |
| Kept in higher precision | Vision tower, router gates, and lm_head |
The vision tower is intentionally left unquantized to preserve image understanding quality.
Usage
Serve with vLLM (OpenAI-compatible API)
docker run --rm --gpus all -p 8000:8000 \
-v "$PWD:/models" \
vllm/vllm-openai:latest \
--model /models/Qwen3.5-4B-AWQ \
--served-model-name Qwen3.5-4B-AWQ \
--quantization compressed-tensors \
--dtype float16 \
--max-model-len 32768
Or directly from the Hub:
docker run --rm --gpus all -p 8000:8000 \
vllm/vllm-openai:latest \
--model sanskar003/Qwen3.5-4B-AWQ \
--quantization compressed-tensors \
--dtype float16
Query it (text)
curl -s http://localhost:8000/v1/chat/completions \
-H 'Content-Type: application/json' -d '{
"model": "Qwen3.5-4B-AWQ",
"messages": [{"role": "user", "content": "Give me three uses for an RTX 5090."}]
}'
Query it (image + text)
curl -s http://localhost:8000/v1/chat/completions \
-H 'Content-Type: application/json' -d '{
"model": "Qwen3.5-4B-AWQ",
"messages": [{"role": "user", "content": [
{"type": "text", "text": "Describe this image."},
{"type": "image_url", "image_url": {"url": "http://images.cocodataset.org/train2017/000000231895.jpg"}}
]}]
}'
Requirements
- A GPU with INT4 Marlin kernel support (vLLM selects it automatically for
compressed-tensorscheckpoints). Verified on NVIDIA Blackwell (sm_120). - A recent vLLM build that supports the
Qwen3_5ForConditionalGenerationarchitecture.
License
Released under Apache 2.0, inherited from the base model Qwen/Qwen3.5-4B. Please review and comply with the base model's license terms.
Citation
Please cite the original Qwen work — see the base model card for citation details.