Qwen3.8-27B W4A16 (AutoRound)
A 4-bit weight-only quant of MuXodious/Qwen3.8-27B-absolute-heresy,
packed in the compressed-tensors format and tuned for single-GPU inference on
a 24 GB card (RTX 3090).
- Parameters: ~28 B (27.99 B logical; base model is 27B-class)
- On disk: 15.79 GB (14.71 GiB)
- Weights in VRAM: ~14.71 GiB — leaving room for the KV cache, activations, and CUDA graphs on a 24 GB card
Quantized with the script in
syv-ai/qwen38-27b-rtx3090 → single-user.
Quantization
- Scheme: W4A16 — 4-bit integer weights, 16-bit activations
- Method: AutoRound, symmetric,
group_size=128,pack-quantized - Format:
compressed-tensors(quant_method: compressed-tensors) - Vision tower: left in bf16 (visual blocks are in the quant
ignorelist), so image understanding is preserved
Serving with vLLM
This is a multimodal (text + image) model. It also ships MTP draft weights, so you can run multi-token prediction for speculative decoding.
Quick start
vllm serve Starw1/Qwen3.8-27B-absolute-heresy-W4A16 \
--gpu-memory-utilization 0.97 \
--max-model-len 140000 \
--kv-cache-dtype fp8 \
--speculative-config '{"method":"mtp","num_speculative_tokens":5}' \
--limit-mm-per-prompt '{"image":4,"video":0}'
Links
- Source model: https://huggingface.co/MuXodious/Qwen3.8-27B-absolute-heresy
- Quantization script: https://github.com/syv-ai/qwen38-27b-rtx3090/tree/main/single-user
Quantized weights only — see the source model card for architecture, training, and license details.