multilingual-e5-small — GGUF (Q8_0)
GGUF conversion of intfloat/multilingual-e5-small, quantized to Q8_0, for on-device retrieval in Offline AI: Private Chat.
- 384 dimensions, 12 layers, 512 max tokens
- 126 MB (f16 is 236 MB)
- Runs via llama.cpp / llama.swift with
embeddings = trueand mean pooling
Quantization is free here
Q8_0 matches f16 to three decimal places on cosine similarity, and produces an identical ranking order — which is what retrieval actually depends on.
| Pair | Q8_0 | f16 |
|---|---|---|
| EN paraphrase | 0.868 | 0.868 |
| EN → FR translation | 0.896 | 0.896 |
| EN → ES translation | 0.843 | 0.843 |
| query → correct passage | 0.848 | 0.848 |
| EN → unrelated topic | 0.737 | 0.736 |
| query → wrong passage | 0.671 | 0.671 |
Note the high floor: unrelated text still scores 0.737. Rank by top-k; do not use an absolute similarity threshold.
Usage
E5 is asymmetric — prefix stored text with passage: and queries with query: . Omitting
the prefixes degrades retrieval, and mixing them is worse than using neither.
Conversion
convert_hf_to_gguf.py fails on this model with BPE pre-tokenizer was not recognized: it
declares architectures: ["BertModel"] but ships an XLM-RoBERTa SentencePiece tokenizer.
Rewrite config.json to architectures: ["XLMRobertaModel"] first, then convert and quantize.
License
MIT, inherited from the base model. Copyright belongs to the original authors — see intfloat/multilingual-e5-small.