Qwen3.5-9B-NVFP4
This is an NVFP4 (4-bit floating point) quantized version of
Qwen/Qwen3.5-9B, packaged in the
compressed-tensors format for efficient inference with
vLLM.
NVFP4 is a Blackwell-native FP4 format and is designed to run on the FP4 tensor cores of NVIDIA Blackwell GPUs (e.g. RTX 5090, sm_120) for high throughput, while preserving the base model's vision-language capabilities.
About the base model
Qwen3.5-9B is a multimodal vision-language model from the Qwen team (Alibaba Group). It accepts interleaved image + text (and video) input and generates text, with a "thinking" reasoning mode enabled by default.
| Base model | Qwen/Qwen3.5-9B |
| Parameters | ~9B |
| Modality | Image-Text-to-Text (vision-language) |
| Architecture | Hybrid Gated DeltaNet + sparse MoE, with a vision encoder |
| Context length | 262,144 tokens native (extensible with RoPE scaling) |
| Languages | 201 languages and dialects |
| License | Apache 2.0 |
For full details on capabilities, benchmarks, and prompt formatting, see the base model card.
Quantization details
| Method | NVFP4 (4-bit floating point) |
| Format | compressed-tensors |
| Quantized modules | Linear layers of the language model |
| Kept in higher precision | Vision tower, router gates, and lm_head |
| Recommended hardware | NVIDIA Blackwell (FP4 tensor cores, sm_120) |
The vision tower is intentionally left unquantized to preserve image understanding quality.
Usage
Serve with vLLM (OpenAI-compatible API)
docker run --rm --gpus all -p 8000:8000 \
-v "$PWD:/models" \
vllm/vllm-openai:latest \
--model /models/Qwen3.5-9B-NVFP4 \
--served-model-name Qwen3.5-9B-NVFP4 \
--quantization compressed-tensors \
--max-model-len 32768
Or directly from the Hub:
docker run --rm --gpus all -p 8000:8000 \
vllm/vllm-openai:latest \
--model sanskar003/Qwen3.5-9B-NVFP4 \
--quantization compressed-tensors
Query it (image + text)
curl -s http://localhost:8000/v1/chat/completions \
-H 'Content-Type: application/json' -d '{
"model": "Qwen3.5-9B-NVFP4",
"messages": [{"role": "user", "content": [
{"type": "text", "text": "Describe this image."},
{"type": "image_url", "image_url": {"url": "http://images.cocodataset.org/train2017/000000231895.jpg"}}
]}]
}'
Requirements
- An NVIDIA Blackwell GPU with FP4 tensor cores (sm_120) for the intended
performance. vLLM selects the FP4 kernel automatically for this
compressed-tensorscheckpoint. - A recent vLLM build that supports the
Qwen3_5ForConditionalGenerationarchitecture.
License
Released under Apache 2.0, inherited from the base model Qwen/Qwen3.5-9B. Please review and comply with the base model's license terms.
Citation
Please cite the original Qwen work — see the base model card for citation details.