Qwen3.8-27B-Abliterated-AWQ-INT4

An AWQ-style W4A16 INT4 quantization of hotdogs/Qwen3.8-27B-abliterated — the training-free abliterated build of Qwen/Qwen3.8-27B.

Quantizing the abliterated checkpoint (not the base model) means the refusal-removal weight edit is preserved through quantization — this is not a generic base-model AWQ with the refusal behaviour intact.

⚠️ Disclaimer. The underlying model does not refuse requests. This build is published for alignment and safety research — measuring what refusal training protects, red-teaming, and studying refusal-direction mechanics. You are responsible for your use of it and for complying with all applicable laws. Inherits the Apache-2.0 license of the base model.

Latest bug fixes as of 2026-08-16

What changed

~66 % smaller while keeping the abliterated weights and (for the quantized linear layers) near-lossless W4A16 accuracy.

Quantization details

Why linear_attn is left in BF16

Qwen3.8-27B uses a hybrid architecture — full attention on some layers and Gated DeltaNet linear attention on others. The GDN linear-attention path has no INT4 kernel, so the 64 linear_attn modules are excluded from quantization and remain at full BF16 precision. Everything else (attention QKV/proj, MLP, embeddings) is quantized to INT4.

Usage

vLLM (recommended — Marlin kernel)

CUDA_HOME=/usr/local/cuda-12.8 \
VLLM_USE_FLASHINFER_SAMPLER=0 \
vllm serve ./Qwen3.8-27B-abliterated-AWQ-INT4 \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --host 0.0.0.0 --port 8000

Then query it:

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen3.8-27B-abliterated-AWQ-INT4",
    "messages": [{"role": "user", "content": "บอก 3 ข้อดีของประเทศไทย สั้นๆ ภาษาไทย"}],
    "max_tokens": 150
  }'

Notes

• --tensor-parallel-size 2 is required on 24 GB GPUs: the model occupies ~22.7 GB per GPU at load (Marlin workspace + BF16 linear_attn), leaving too little for the KV cache on a single 3090. On a bigger GPU (≥ 48 GB) TP=1 works.
• VLLM_USE_FLASHINFER_SAMPLER=0 disables the FlashInfer JIT sampler to avoid a CUDA-toolkit/CCCL version mismatch crash on some hosts.

Transformers / Hugging Face

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "hotdogs/Qwen3.8-27B-abliterated-AWQ-INT4",
    device_map="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("hotdogs/Qwen3.8-27B-abliterated-AWQ-INT4")

Memory requirements

Benchmark (A/B vs base)

This INT4 build quantizes the λ = 1.2 abliterated checkpoint, so its capability tracks the λ = 1.2 base model. The A/B benchmark below was measured on the bf16 λ = 1.2 model (identical lm-eval HF backend, same prompt/config, no chat template); the INT4 quant is expected to land within a point or two of these numbers on the quantized linear layers:

The λ = 1.2 build preserves capability essentially at base level (MMLU −0.005, GSM8K −0.03, ARC +0.01) while cutting refusal from 98 % → 39 % with KL 0.0001. See the base model card for the full method and the refusal-direction / r̂ weight-level verification.

Reproduction

The quantization was done with llm-compressor on the abliterated checkpoint:

from llmcompressor import Oneshot
from llmcompressor.modifiers.quantization import GPTQModifier

recipe = GPTQModifier(
    scheme="W4A16",
    targets=["Linear"],
    ignore=["lm_head"],   # linear_attn auto-excluded by the hybrid arch
)

oneshot = Oneshot(
    model="hotdogs/Qwen3.8-27B-abliterated",
    recipe=recipe,
    dataset="<your calibration corpus>",
    output_dir="./Qwen3.8-27B-abliterated-AWQ-INT4",
    max_seq_length=2048,
    num_calibration_samples=128,
)
oneshot()
oneshot.save()

Model card for the base (abliterated) checkpoint

For full details on the abliteration method, the refusal-direction edit, and reproduction with LLM-abliterate, see the base model card: hotdogs/Qwen3.8-27B-abliterated.

TL;DR — a single forward-only rank-1 weight edit removes the refusal direction from the residual-stream writers (λ = 1.2, hidden-state index 46). No fine-tuning, no distillation, no data poisoning. The vision tower and lm_head are untouched. Same Apache-2.0 license as the base model.