Qwen3-Embedding-0.6B W8A16 g128, int8-RESIDENT (custom code)
Same int8 weight-only group_size-128 quantization as
Qwen3-Embedding-0.6B-W8A16-g128,
but packaged as repo custom code (trust_remote_code=True) whose Int8GroupLinear modules keep
the int8 weights RESIDENT in VRAM and dequantize one layer at a time inside forward.
Why: transformers decompresses compressed-tensors checkpoints to bf16 in VRAM (lazily, at first
forward) and keeps the bf16, so the stock format saves download but not resident memory. This
packaging saves both.
Measured (H100):
| bf16 | this repo | |
|---|---|---|
| Resident weights (transformers, both after forwards) | 1136 MiB | 740 MiB |
Infinity 0.0.77, --batch-size 8, process VRAM | 2490 MiB | 2102 MiB (-16%) |
| Latency in-process, batch 1 / batch 32 | 17.2 / 23.3 ms | 23.0 / 29.8 ms |
Fidelity: quantization identical to the W8A16-g128 repo (recall@10 0.976 vs bf16 on a 360-doc multi-script index; embedding cosine 0.999 vs the bf16 service).
Usage
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("Nicolassuez/Qwen3-Embedding-0.6B-W8A16-g128-resident",
trust_remote_code=True)
or AutoModel.from_pretrained(..., trust_remote_code=True) with last-token pooling + L2 normalize.
Works in Infinity:
infinity_emb v2 --model-id Nicolassuez/Qwen3-Embedding-0.6B-W8A16-g128-resident \
--trust-remote-code --no-bettertransformer --device cuda
(--no-bettertransformer is required: Infinity 0.0.77's BetterTransformer probe crashes when
optimum is absent or >= 2.0.)
nn.Module.to(dtype) only casts floating-point tensors, so the int8 buffers survive the
.to(float16/bfloat16) calls sentence-transformers and Infinity make.
Caveat: transformers version pinning
Qwen3 forward numerics drift between transformers 4.57 and 5.14 (embedding cosine ~0.973 between the two, measured on this model; the drift exists for the bf16 base too). Embeddings indexed with one version should be queried with the same version.
Update: int8 embedding table
The embedding table (151669 x 1024, 26% of parameters) is now ALSO stored int8, with one bf16 scale per row (a lookup gathers single rows, no accumulation crosses the quantized axis, so per-row scaling is the natural granularity). Resident weights drop 740 -> 592 MiB. Validated on the 360-doc / 70-query benchmark against bf16: mean weight cosine unchanged (0.99922 vs 0.99923 with the bf16 table) and every top-1 change happens on candidate pairs the bf16 model itself separates by less than 0.01. Do NOT replicate this on the reranker: its lm_head is tied to the table and the yes/no score reads it directly.