Qwen2.5-VL-3B-Instruct — GGUF Quantizations (VLM)

Quantized GGUF versions of Qwen/Qwen2.5-VL-3B-Instruct

This is a Vision-Language Model (VLM) — it can understand both text and images.

Works with llama.cpp · LM Studio · Jan · Ollama

Quantized by Dhptl on June 11, 2026 using quant-kit

[!IMPORTANT] This VLM requires TWO files — a text backbone GGUF and the mmproj vision encoder GGUF. Download one text backbone (e.g. Q4_K_M) and the mmproj file. Both must be in the same folder.

📦 Available Files

🔤 Text Backbone (quantized — pick ONE)

🖼️ Vision Encoder — mmproj (always required, always F16)

⚠️ You need BOTH files — one text backbone + the mmproj — to run this VLM.

⚡ Speed Benchmarks

Run python benchmark.py --model Qwen2.5-VL-3B-Instruct to generate results.

🚀 How to Use

LM Studio (Easiest — GUI)

1. Search for Dhptl/Qwen2.5-VL-3B-Instruct in LM Studio
2. Download the Q4_K_M text file and the mmproj file
3. Load the model — LM Studio automatically uses both files

Ollama

ollama run dhptl/qwen2.5-vl-3b-instruct

llama.cpp CLI — Text + Image

# Download both files to the same directory, then:
./llama-llava-cli \
  -m Qwen2.5-VL-3B-Instruct-Q4_K_M.gguf \
  --mmproj Qwen2.5-VL-3B-Instruct-mmproj-f16.gguf \
  --image /path/to/your/image.jpg \
  -p "Describe this image in detail." \
  -n 512

llama.cpp CLI — Text only (no image)

./llama-cli \
  -m Qwen2.5-VL-3B-Instruct-Q4_K_M.gguf \
  -p "You are a helpful assistant." \
  --conversation

Python — llama-cpp-python

from llama_cpp import Llama
from llama_cpp.llama_chat_format import Llava16ChatHandler

# Load VLM with mmproj
chat_handler = Llava16ChatHandler(clip_model_path="./Qwen2.5-VL-3B-Instruct-mmproj-f16.gguf")
llm = Llama(
    model_path="./Qwen2.5-VL-3B-Instruct-Q4_K_M.gguf",
    chat_handler=chat_handler,
    n_gpu_layers=-1,
    n_ctx=4096,
    logits_all=True,
)

# Text + image inference
response = llm.create_chat_completion(
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}},
                {"type": "text",      "text":      "What do you see in this image?"}
            ]
        }
    ]
)
print(response["choices"][0]["message"]["content"])

🔍 VLM Architecture

This model uses a two-component architecture:

Why is mmproj always F16? The vision encoder maps image pixels to token embeddings. Quantizing it causes visible visual artifacts and degraded image understanding. It stays at F16 (half precision) which is already very efficient at ~1-2GB for most models.

🔍 About GGUF Quantization

💬 Community & Feedback

Found an issue? Open a Discussion in the Community tab.

If useful, please:

• ⭐ Star quant-kit on GitHub
• 👍 Like this model on HuggingFace