Qwen3.8-27B (W4A16 Quantized via AutoRound)
This repository contains a W4A16 (4-bit weights, 16-bit activations) quantized version of Qwen/Qwen3.8-27B generated using Intel's AutoRound algorithm.
⚡ Quantization Details
The model was calibrated and quantized using production-grade settings to minimize accuracy degradation while significantly lowering VRAM requirements:
- Quantization Algorithm: AutoRound
- Bits / Precision: W4A16 (4-bit integer weights, 16-bit activation)
- Group Size: 64 (provides higher reconstruction fidelity than standard 128)
- Symmetric (
sym):True - Calibration Samples (
nsamples): 512 - Tuning Iterations (
iters): 600 - Vision Tower (
quant_nontext_module):False(Kept in BF16 to preserve visual reasoning and OCR precision) - Special Modules (
layer_config): Multi-Token Prediction (mtp,mtp.fc) layers preserved in nativebfloat16.
🚀 Usage & Quickstart
1. Inference via vLLM
For high-throughput production serving:
vllm serve Vishva007/Qwen3.8-27B-W4A16-AutoRound \
--quantization auto-round \
--dtype bfloat16 \
--max-model-len 4096 \
--gpu-memory-utilization 0.90
(Note: For the GPTQ format repo, you can set --quantization gptq if required by your backend).
📊 VRAM & Performance Benefits
- Original Model (BF16): ~54 GB VRAM required (needs 2x A100/A6000 or 80GB VRAM)
- Quantized Model (W4A16 Group 64): ~16–18 GB VRAM (can easily run on a single RTX 3090 / 4090 / A5000 24GB)
- Speedup: Reduced memory bandwidth bottleneck leading to faster decoding token speeds.
🚀 Deploy on RunPod
One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.
🎁 Need GPU compute? Sign up via RunPod and get $5–$500 in free credits when you add your first $10.
PyTorch 2.14
| Template | CUDA Version | Docker Image | Template ID | Deploy |
|---|---|---|---|---|
| PyTorch 2.14 (CUDA 12.6) | 12.6 | vishva123/cuda-12.6-pytorch-2.14-runpod | d7lxsa4w9m |  |
| PyTorch 2.14 (CUDA 13.0) | 13.0 | vishva123/cuda-13.0-pytorch-2.14-runpod | yk0y6j6rpg | |
| PyTorch 2.14 (CUDA 13.2) | 13.2 | vishva123/cuda-13.2-pytorch-2.14-runpod | gsp4gwx0nw | |
PyTorch 2.13
| Template | CUDA Version | Docker Image | Template ID | Deploy |
|---|---|---|---|---|
| PyTorch 2.13 (CUDA 12.6) | 12.6 | vishva123/cuda-12.6-pytorch-2.13-runpod | gmlupxnxfk | |
| PyTorch 2.13 (CUDA 13.0) | 13.0 | vishva123/cuda-13.0-pytorch-2.13-runpod | y3j8xvk4f4 | |
| PyTorch 2.13 (CUDA 13.2) | 13.2 | vishva123/cuda-13.2-pytorch-2.13-runpod | vigpissn5w | |
PyTorch 2.12
| Template | CUDA Version | Docker Image | Template ID | Deploy |
|---|---|---|---|---|
| PyTorch 2.12 (CUDA 12.6) | 12.6 | vishva123/cuda-12.6-pytorch-2.12-runpod | ctmz86zmf0 | |
| PyTorch 2.12 (CUDA 13.0) | 13.0 | vishva123/cuda-13.0-pytorch-2.12-runpod | qjko5yiwzi | |
| PyTorch 2.12 (CUDA 13.2) | 13.2 | vishva123/cuda-13.2-pytorch-2.12-runpod | ifg6xmye0f | |
📚 Acknowledgments
- Quantization performed using Intel AutoRound.
- Base architecture provided by the Qwen Team.