Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-heretic-W4A16
W4A16 (int4) quantization of
gorbatjovy/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-heretic —
the Heretic-abliterated build of
DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1.
Runs the full vision-language model with MTP / DFlash2 speculative decoding on a single 24 GB GPU (e.g. RTX 4090). ~19 GB on disk. Credit for all capability goes to DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1; this repo only removes refusals and quantizes.
⚠️ Disclaimer — read before use
This model has had its safety alignment removed via abliteration. It will comply with harmful requests the base model would refuse. Released for research (interpretability, red-teaming, robustness). You own what you do with it and what it generates. Add your own moderation before any deployment. Use must comply with the inherited Apache 2.0 license and applicable law. No liability is accepted for misuse.
Quantization
Produced with AutoRound, served with compressed-tensors in vLLM.
| Component | Precision |
|---|---|
Language-decoder Linear layers | int4, group 128, symmetric (W4A16) |
lm_head, embed_tokens, MTP module | int8, group 128 |
Vision tower (visual.*), Gated DeltaNet in_proj_a/in_proj_b | BF16 (untouched) |
The vision tower and the recurrence controls are kept at BF16 so image understanding and the hybrid-attention dynamics are unaffected. Round-trip errors on the int8 steps are 0.6–1.5%.
Refusal / quality (served build)
Measured on this quantized build, served under vLLM with an opening-phrase refusal classifier:
| Benchmark | Refusal |
|---|---|
| AdvBench | 0.0% |
| JailbreakBench (harmful) | 0.0% |
| MaliciousInstruct | 0.0% |
The underlying abliteration scored 8 / 100 on Heretic's own (stricter) keyword scorer at KL divergence 0.0315 vs the base — see the BF16 card for method details. Vision verified working on the quantized build.
Quantization loss vs BF16
Measured against the full-precision
BF16 build
with the same harness hitting both (perplexity via vLLM prompt_logprobs, so it is a
true like-for-like measurement, not two different loaders):
| Metric | BF16 | W4A16 | Cost of int4 |
|---|---|---|---|
| WikiText-2 perplexity | 7.537 | 7.660 | +1.6% |
| GSM8K (100 problems, matched basis) | 95% | 93% | −2 pts |
Tool calling (12 cases, qwen3_xml parser) — valid / correct tool / correct args | 11 / 11 / 11 | 10 / 10 / 10 | −1 (noise at n=12) |
The int4 quantization is close to free: language modeling barely moves, tool-calling is preserved, and the only real cost is a ~2-point dip on GSM8K — expected, since multi-step arithmetic is the most quantization-sensitive thing to measure. Vision is verified working on the quantized build.
Serving (vLLM)
vllm serve gorbatjovy/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-heretic-W4A16 \
--served-model-name qwen3.8-27b \
--max-model-len 32768 \
--trust-remote-code \
--reasoning-parser qwen3 \
--limit-mm-per-prompt '{"image":1}'
Measured ~142 tok/s single-stream on an RTX 4090 with DFlash2 speculative decoding (mean acceptance length ~3.1). MTP speculative decoding also works out of the box.
License
Apache 2.0, inherited. Abliteration and quantization do not change the license.