Qwen3.8-27B — BenQ 6-bit
Qwen3.8-27B quantized with BenQ, a per-row lookup-table scheme, at 6.16 bits/weight. On HumanEval it is statistically indistinguishable from the Q8_0 GGUF of the same model while using 28% fewer bits.
The vision tower is included and quantized.
What BenQ stores
Each weight is one 6-bit code: 1 sign bit + 5 magnitude-tier bits. The code indexes
that output row's own table of 64 signed magnitudes (32 negative, ascending, then 32
positive). Decode is exactly lut.gather(1, codes), and the included Triton kernel fuses
that gather into the GEMM so the dequantized matrix is never materialised.
| component | bits/weight |
|---|---|
| code | 6.000 |
| per-row LUT (64 bf16 entries, amortised over 5120–17408 columns) | 0.164 |
| total | 6.16 |
This is not a GALE format — no rotation, no outlier side-channel.
Results
HumanEval pass@1, greedy, identical harness/prompts/scorer for both arms, on 2×RTX 3090:
| model | bits/weight | pass@1 | tok/s |
|---|---|---|---|
| BenQ 6-bit (this repo) | 6.16 | 131/164 = 0.799 | 6.8 |
| Q8_0 GGUF (llama.cpp) | 8.50 | 132/164 = 0.805 | 6.0 |
A one-problem difference. With n=164 the standard error is ≈3.7 points, so these are the same score; the claim is parity at 28% fewer bits, not a win.
Both arms were re-scored after trimming each completion to its longest syntactically valid prefix. Without that step the naive stop-marker heuristic truncated the more verbose Q8_0 completions mid-docstring and scored 36 of them as syntax failures, producing a spurious 15-point gap. Reported here is the corrected, symmetric number.
Reconstruction error (output-space relative error on held-out activations):
| group | out_relerr | calibration |
|---|---|---|
| 64 decoder layers (496 linears) | 0.0114 | 8192 real tokens, GPTQ alternation |
| vision tower (110 linears) | 0.0078 | 256 real photographs |
lm_head | 0.0084 | real hidden states from the quantized layer stack |
embed_tokens | 0.0330 | uncalibrated — see below |
embed_tokens cannot be calibrated: its input is a one-hot token id, so no meaningful
Hessian exists. It falls back to nearest-level assignment against the Lloyd-refined table.
Stated rather than papered over.
Why equal-width shells
The level placement was chosen empirically on 62 real matrices from this model, measured in output space with the bit-count confound controlled:
| placement | tiers | bits | out_relerr |
|---|---|---|---|
| quantile (equal population) | 8 | 4 | 0.0530 |
| recursive-mean | 8 | 4 | 0.0492 |
| equal-width shells | 8 | 4 | 0.0467 |
| quantile | 32 | 6 | 0.0245 |
| recursive-mean | 32 | 6 | 0.0173 |
| equal-width shells | 32 | 6 | 0.0141 |
At matched 6 bits, shells beat quantile placement by 42.7% and recursive-mean by 18.5%, winning on 62/62 matrices. Equal-width bands make the population deliberately skewed — the innermost shells hold most of the weights while the sparse tail gets its own narrow bands — which is the opposite of the usual equal-population default.
Precedent, flagged honestly: per-row non-uniform LUTs resemble SqueezeLLM and LUT-GEMM; recursive-mean initialisation is LBG binary splitting (1980). The shell-placement result is the contribution here.
Usage
from benq.load_benq_qwen import load_benq_qwen
model, tok = load_benq_qwen('/path/to/this/repo') # splits across 2 GPUs by capacity
ids = tok("def fibonacci(n):\n", return_tensors='pt').input_ids.to(model.device)
print(tok.decode(model.generate(ids, max_new_tokens=64)[0]))
benq/server.py serves an ollama-native /api/chat (streaming, tools, thinking) for
drop-in use with ollama clients.
Requires ~28 GB of VRAM as stored (codes are raw uint8; the information is 6.16
bits, so packing to 6-bit would bring this to ~21 GB). Two 24 GB cards work; one does not.
transformers>=5.5.0 for qwen3_5, plus triton and accelerate.
Multi-token prediction head
mtp_bf16.safetensors contains the model's multi-token-prediction head — 15 tensors,
425M params, kept in bf16 rather than quantized. It is included because it is the only
route past the memory-bandwidth ceiling on batch-1 decode: reading 18.8 GB of weights per
token caps a 2×3090 at roughly 50 tok/s no matter how good the kernel is, and accepting
2–3 speculative tokens per weight-read amortises that.
It is shipped, not used: load_benq_qwen does not currently load it and the server does
no speculative decoding. Kept in bf16 because at 3% of the artifact the size is not worth
the calibration question — its input is a hidden state concatenated with an embedding,
which is not a distribution the layer-wise calibration path captures.
Limitations
- No WikiText-2 / C4 perplexity, and no matched-bpw comparison against GPTQ or AWQ. The comparison here is against Q8_0 only. Treat cross-paper comparisons as unsupported.
- HumanEval n=164 cannot resolve differences under ~2 points.
embed_tokensis uncalibrated (above).- Codes ship unpacked, so the on-disk size is 8 bits/weight rather than the 6.16 the format actually needs.
- The
mtphead ships unquantized (bf16, 0.85 GB) and is not wired into generation — see below.