Qwen3-Embedding-4B MLX 8-bit
MLX 8-bit quantization of Qwen/Qwen3-Embedding-4B, produced with mlx-embeddings on Apple Silicon.
What is this?
Qwen3-Embedding is a decoder-only LLM-style text embedding model from the Qwen3 family, using last-token pooling to produce dense vector representations. It scores near the top of MMTEB multilingual benchmarks while retaining Apache-2.0 licensing.
Quantization
- Method: MLX affine quantization (
mlx_embeddings.convert), group_size=64 - Bits per weight: 8
- Output size: 4.0 GB (vs ~7.5 GB for bf16 source)
Quickstart
from mlx_embeddings import load
model, tokenizer = load("majentik/Qwen3-Embedding-4B-MLX-8bit")
inputs = tokenizer(
["What is the capital of France?", "Paris is the capital of France."],
padding=True, truncation=True, return_tensors="mlx"
)
outputs = model(inputs["input_ids"], attention_mask=inputs["attention_mask"])
embeddings = outputs.text_embeds # already L2-normalised, shape [batch, dim]
For sentence similarity:
import mlx.core as mx
e = embeddings
scores = (e[0] @ e[1:].T).tolist()
print(scores)
Model Specifications
| Property | Value |
|---|---|
| Base Model | Qwen/Qwen3-Embedding-4B |
| Architecture | Decoder-only (Qwen3ForCausalLM) with last-token pooling |
| Parameters | 4B (4.0B) (pre-quantization) |
| Context Length | 32K |
| Embedding Dim | 2560 |
| BF16 Size | ~7.5 GB |
| License | apache-2.0 |
| Languages | 100+ (multilingual) |
License
Apache 2.0 — inherited from the upstream Qwen3-Embedding model. Free for research and commercial use.
See also
- Base: Qwen/Qwen3-Embedding-4B
- Official GGUF: Qwen/Qwen3-Embedding-4B-GGUF (if published by Qwen)
- mlx-embeddings package: https://github.com/Blaizzy/mlx-embeddings
- Garden hub: majentik/garden
- MTEB leaderboard: https://huggingface.co/spaces/mteb/leaderboard