Qwen3.5-9B VLM Q4_K_M GGUF

Ready-to-run Vision Language Model — bundled Q4_K_M quantization + multimodal projector for immediate VLM inference with llama.cpp.

Overview

This repository provides a complete, self-contained VLM package for Qwen3.5-9B in GGUF format. Unlike text-only GGUF repos, this includes the multimodal projector (mmproj) required for vision capabilities — no need to hunt for separate files.

Total VRAM/RAM required: ~7–8 GB for full VLM inference

Key Features

• Natively Multimodal — Qwen3.5 uses early fusion architecture where vision and language are jointly trained from the ground up, not bolted on as an afterthought
• Outperforms Qwen3-VL — Cross-generational improvements across visual understanding, reasoning, coding, and agent benchmarks
• 262K Context Length — Native 262,144 token context window, extensible to 1M+ tokens
• 201 Languages — Broad multilingual support including CJK, European, Arabic, Southeast Asian, and more
• Efficient Quantization — Q4_K_M provides an excellent balance of quality and size, retaining strong performance while fitting in ~6 GB

Architecture

Qwen3.5 represents a paradigm shift from the separate "LLM + Vision Encoder" approach (used by Qwen3-VL, LLaVA, etc.) to a unified early-fusion architecture:

Image → Vision Encoder → Multimodal Projector (mmproj) ─┐
                                                         ├→ Unified Transformer → Output
Text → Token Embedding ─────────────────────────────────┘

The vision encoder processes images into visual tokens that are projected into the same embedding space as text tokens via the mmproj file, enabling seamless multimodal reasoning within a single transformer.

Quickstart

Prerequisites

• llama.cpp (built from source or pre-built binaries)
• ~8 GB available RAM/VRAM

Download

# Download both files
huggingface-cli download jc-builds/Qwen3.5-9B-VLM-Q4_K_M-GGUF \
  --local-dir Qwen3.5-9B-VLM-Q4_K_M-GGUF

Run with llama.cpp CLI

# Image understanding
./llama-cli \
  --model Qwen3.5-9B-VLM-Q4_K_M-GGUF/Qwen3.5-9B-Q4_K_M.gguf \
  --mmproj Qwen3.5-9B-VLM-Q4_K_M-GGUF/mmproj-F16.gguf \
  --image your_image.jpg \
  -p "Describe what you see in this image in detail." \
  --temp 0.7 --top-p 0.9 \
  -n 512

# Text-only mode (omit --mmproj and --image)
./llama-cli \
  --model Qwen3.5-9B-VLM-Q4_K_M-GGUF/Qwen3.5-9B-Q4_K_M.gguf \
  -p "Explain quantum entanglement simply." \
  --temp 0.7 \
  -n 512

Run with llama.cpp Server

./llama-server \
  --model Qwen3.5-9B-VLM-Q4_K_M-GGUF/Qwen3.5-9B-Q4_K_M.gguf \
  --mmproj Qwen3.5-9B-VLM-Q4_K_M-GGUF/mmproj-F16.gguf \
  --port 8080

Then send requests with images via the OpenAI-compatible API:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.5-9b-vlm",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "text", "text": "What is in this image?"},
          {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}
        ]
      }
    ]
  }'

Use with iOS / On-Device (llama.cpp Swift bindings)

This model is optimized for on-device inference. Load both files in your llama.cpp-based iOS app:

// Load main model
let modelParams = llama_model_default_params()
let model = llama_model_load_from_file("Qwen3.5-9B-Q4_K_M.gguf", modelParams)

// Load vision projector
let clipModel = clip_model_load("mmproj-F16.gguf", /* verbosity */ 1)

Quantization Details

Q4_K_M uses a mixed quantization strategy where more important layers retain higher precision, providing a strong quality-to-size ratio. The vision projector is kept at FP16 to preserve visual understanding fidelity.

Compatibility

Benchmarks

Qwen3.5-9B delivers exceptional performance across vision, text, and reasoning tasks — often matching or exceeding models many times its size.

Note: Quantization to Q4_K_M typically results in minimal degradation (~1-2%) compared to FP16 for models of this size. All scores below are from the FP16 base model.

Vision & Multimodal Performance

Capability Radar

Text & Reasoning Performance

Credits & Acknowledgments

• Base Model: Qwen Team, Alibaba Cloud
• mmproj Extraction: Derived from unsloth/Qwen3.5-9B-GGUF
• Quantization Format: GGML / llama.cpp

License

This model is released under the Apache 2.0 License, consistent with the base Qwen3.5-9B model.

Citation

@article{qwen3.5,
  title={Qwen3.5 Technical Report},
  author={Qwen Team},
  year={2025},
  url={https://qwen.ai/blog?id=qwen3.5}
}