Qwen3-VL-Embedding-8B GGUF — Quantized by BatiAI
GGUF quantizations of Qwen/Qwen3-VL-Embedding-8B — the quality tier of Qwen3's vision-language embedding family. Part of BatiAI's on-device RAG stack for BatiFlow.
When to pick 8B over 2B?
| Use case | Pick |
|---|---|
| Workstation / desktop Mac, retrieval quality matters | 8B — richer embeddings, better semantic separation |
| Laptop / Mac mini / latency matters | 2B — 4× smaller, sufficient for most tasks |
What does it do?
VL (Vision-Language) embedding turns either text OR images into dense vectors in the same embedding space. This enables:
- Search photos by text — "beach sunset" retrieves matching photos without manual tagging
- Search text by image — drop a screenshot, find similar notes
- Cross-modal RAG — index PDFs, notes, and images together in one vector DB
Quick Start
Text embedding (Ollama)
ollama pull batiai/qwen3-vl-embed-8b:q8
curl http://localhost:11434/api/embeddings -d '{
"model": "batiai/qwen3-vl-embed-8b:q8",
"prompt": "What is the capital of France?"
}'
Returns a 3584-dim float vector (8B model uses wider embedding than 2B's 2048).
Image embedding
Image support requires llama.cpp's mtmd multimodal build. See upstream docs.
Available Quantizations
| File | Quant | Size | Recommended |
|---|---|---|---|
Qwen3-VL-Embedding-8B-Q6_K.gguf | Q6_K | 5.8 GB | balanced |
Qwen3-VL-Embedding-8B-Q8_0.gguf | Q8_0 | 7.5 GB | near-lossless, best for retrieval quality |
Embedding models are sensitive to low-bit quantization (vector quality drops). Q6_K minimum.
Quality note
Direct embedding-quality evaluation (e.g. MTEB retrieval) is more involved than pairwise reranker testing and takes longer to run locally — it will be added as measured. In the meantime, our sibling Qwen3-Reranker-8B card shows Q6_K ↔ Q8_0 Pearson correlation r = 0.9986 on 40 hard-negative triples. Same architecture family, same quantization approach — we expect this embedding model to behave similarly (quantization drift under noise floor).
Why Qwen3-VL-Embedding?
- SOTA on MTEB — top multilingual embedding model
- Multilingual — en / ko / ja / zh
- Multimodal — text and image in the same embedding space
Why BatiAI?
- Quantized directly from Alibaba's BF16 safetensors
- BatiAI-signed metadata
- Part of a full on-device RAG stack
Technical Details
- Original Model: Qwen/Qwen3-VL-Embedding-8B
- Architecture: Qwen3-VL with pooling head
- Parameters: 8 B (text tower) + vision tower
- License: Apache 2.0
BatiAI's RAG Stack
| Role | Model | HF |
|---|---|---|
| VL Embedding (2 B) | Qwen3-VL-Embedding-2B | batiai/Qwen3-VL-Embedding-2B-GGUF |
| VL Embedding (8 B) | Qwen3-VL-Embedding-8B | this repo |
| Reranker (0.6 B) | Qwen3-Reranker-0.6B | batiai/Qwen3-Reranker-0.6B-GGUF |
| Reranker (4 B) | Qwen3-Reranker-4B | batiai/Qwen3-Reranker-4B-GGUF |
| Reranker (8 B) | Qwen3-Reranker-8B | batiai/Qwen3-Reranker-8B-GGUF |
| Chat LLM (35 B-A3B) | Qwen3.6-35B-A3B | batiai/Qwen3.6-35B-A3B-GGUF |
License
Mirrors upstream Qwen Apache 2.0. Commercial use permitted.