Qwen3-Embedding-0.6B-Q8-NVFP4
Qwen3-Embedding-0.6B · NVFP4 quantized · 507 MB
A hand-built GGUF of the Qwen3-Embedding-0.6B text embedding model, quantized with a mixed NVFP4 recipe for Blackwell GPUs. 1024-dim embeddings, 32K context, last-token pooling.
Credits
- The Qwen team (Qwen/Qwen3-Embedding-0.6B) — built the base model.
- AIconjured — the NVFP4 quantization and packaging.
Files
| File | Size | Description |
|---|---|---|
Qwen3-Embedding-0.6B-NVFP4.gguf | 507 MB | 310 tensors: 84 NVFP4 (FFN GEMMs) + 113 Q8_0 (embeddings, attention, norms) + 113 F32 (norms/biases/ssm) |
Size: 507 MB — down from 1198 MB (F16) and 639 MB (Q8_0).
Quantization recipe
Mixed-precision, built with llama.cpp's --tensor-type-file plus an imatrix generated from ~144 KB of mixed calibration text:
| Type | Tensors | What | Why |
|---|---|---|---|
| NVFP4 | 84 | ffn_gate/up/down (all 28 layers) | The big GEMMs — where the bits are. Native Blackwell FP4 path. |
| Q8_0 | 113 | token_embd, attn_q/k/v/output, all attention norms | Embedding geometry is sensitive; Q8_0 is near-lossless. |
| F32 | 113 | All norm weights, biases | Tiny but quantizing them causes quality loss. |
Recipe (llama.cpp --tensor-type-file format):
token_embd\.weight=q8_0
output_norm\.weight=q8_0
blk\.\d+\.ffn_gate\.weight=nvfp4
blk\.\d+\.ffn_up\.weight=nvfp4
blk\.\d+\.ffn_down\.weight=nvfp4
Measured quality
Embedding cosine similarity vs the F16 original, measured on 20 diverse texts through the actual llama.cpp runtime (last-token pooling, 1024-dim):
| Quant | Mean cosine vs F16 | Min (worst case) |
|---|---|---|
| Q8_0 | 0.99936 | 0.99765 |
| NVFP4 (this model) | 0.97472 | 0.95203 |
Short factual queries stay tight (~0.98); harder semantic queries drift up to ~18°. In retrieval, expect a small recall drop concentrated on nuanced queries. If you need near-lossless embeddings, use the Q8_0 version instead.
Usage
Ollama:
FROM ./Qwen3-Embedding-0.6B-NVFP4.gguf
llama.cpp:
llama-server -m Qwen3-Embedding-0.6B-NVFP4.gguf --embedding -c 32768
# POST /embeddings {"input": "your text"}
Hardware notes
- Built and verified on 2× RTX 5060 Ti (Blackwell, sm_120a) with a CUDA 12.8 llama.cpp build.
- NVFP4 kernels use the native Blackwell FP4 path; on non-Blackwell GPUs the tensors still load and run (dequantized), with less speed advantage.
- The model is tiny (507 MB) — fits entirely in a single GPU with large context headroom.
Reproduction
- llama.cpp:
378aa2ebc(2026-08-27), CUDA build - Quantize:
llama-quantize --imatrix imatrix-emb.gguf --tensor-type-file recipe-emb-nvfp4.txt ... - Imatrix: 32 chunks × 2048 tokens, mixed wiki/prose calibration
License
Apache-2.0, matching the base Qwen/Qwen3-Embedding-0.6B model.