Qwen3-Reranker-0.6B W8A16 g128 GPTQ, int8-RESIDENT (custom code)
The exact same int8 values and scales as
Qwen3-Reranker-0.6B-W8A16-g128-gptq
(bit-exact unpack of the GPTQ solution, zero requantization), repackaged as repo custom code
(trust_remote_code=True) whose Int8GroupLinear modules keep the int8 weights RESIDENT in VRAM
and dequantize per forward, instead of transformers' decompress-to-bf16-and-keep behavior.
Resident weights: 740 MiB instead of 1136 MiB bf16. lm_head stays bf16 (tied to the embedding
table; the yes/no score is read from it). Fidelity is that of the GPTQ parent: identical bf16 top-1
on every query bf16 itself decides (score gap >= 0.01), 70 queries x 20 candidates.
Usage
Load with AutoModelForCausalLM.from_pretrained(..., trust_remote_code=True) and apply the
official Qwen3-Reranker scaffold; score = P("yes") from the last-position logits. Pass
logits_to_keep=1 to avoid materializing full-sequence vocab logits (see the parent model card for
the complete snippet; only the repo id and trust_remote_code=True differ).
Companion embedder: Qwen3-Embedding-0.6B-W8A16-g128-resident.
Caveat: transformers version pinning
Qwen3 forward numerics drift between transformers 4.57 and 5.14 (measured on the embedding twin; applies to the bf16 base too). Keep the serving version pinned.