WeMM-Embedding-2B — Quantization Quality (STS-B)

Evaluation set: STS-B test (1,379 sentence pairs, human similarity 0-5). Baseline: BF16 GGUF run in llama.cpp (same engine as all quants), so the measured difference reflects quantization error only.

Metrics

• STS-B Spearman ρ: rank correlation between model cosine similarities and human similarity scores. Higher is better.
• Δρ vs BF16: relative drop of ρ against the BF16 baseline. Negative means the quant scored slightly above baseline (within noise).
• Emb Cosine vs BF16: mean cosine similarity between each sentence's embedding and its BF16 counterpart (space fidelity). 1.0 = identical.
• Pair Cosine Pearson vs BF16: Pearson correlation of per-pair cosine similarities vs BF16 (ranking fidelity). 1.0 = identical ordering.

Conclusion

• Q8_0, Q6_K, Q5_K_M and IQ4_XS show negligible quality loss (|Δρ| < 0.05%, Emb Cosine > 0.985) and are safe drop-in replacements.
• Q4_K_M (4.85 bpw) shows a small but visible drop (Δρ ≈ +0.60%, Emb Cosine 0.985) — notably worse than the equally-sized IQ4_XS, so prefer IQ4_XS or Q5_K_M over Q4_K_M when size is comparable.
• IQ3_M (3.76 bpw) is the only variant with a clearly measurable drop (Δρ ≈ +0.89%, Emb Cosine 0.93); use only when storage is critical.

Usage (llama.cpp GGUF)

All files here are GGUF and run with llama.cpp. Replace the model file with the quant you downloaded. Use -ngl 999 to offload layers to GPU (omit or -ngl 0 for CPU-only).

Text embedding — command line

llama-embedding \
  -m WeMM-Embedding-2B-Q5_K_M.gguf \
  -p "Represent the meaning of this sentence." \
  --pooling last

Text embedding — HTTP server

llama-server \
  -m WeMM-Embedding-2B-Q5_K_M.gguf \
  --embedding \
  -ngl 999 --host 0.0.0.0 --port 8080

Then request embeddings via the OpenAI-compatible endpoint:

curl http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"input": "Represent the meaning of this sentence.", "model": "WeMM-Embedding-2B-Q5_K_M"}'

Multimodal (image / video) — HTTP server

The visual projector (mmproj-WeMM-Embedding-2B-BF16.gguf) is required for image and video inputs:

llama-server \
  -m WeMM-Embedding-2B-Q5_K_M.gguf \
  --mmproj mmproj-WeMM-Embedding-2B-BF16.gguf \
  --embedding \
  -ngl 999 --host 0.0.0.0 --port 8080

Send image/video inside the chat content the same way as the base model (interleave image/video before text).

Notes

• Output is a 2048-dim L2-normalized vector; matryoshka truncation (e.g. --embd-normalize + slicing) follows the base model's matryoshka_dimensions [64, 128, 256, 512, 1024, 2048].
• Q8_0 / Q4_K_M / BF16 are mirrored from DreamBlooms/WeMM-Embedding-2B-GGUF; Q6_K / Q5_K_M / IQ4_XS / IQ3_M were produced for this repo with llama-quantize from the same BF16 master.