Vishva007/Qwen3.5-9B-W4A16-AutoRound-GPTQ
This is a W4A16 (4-bit weight, 16-bit activation) quantized version of Qwen/Qwen3.5-9B, produced using AutoRound — Intel's sign gradient descent based quantization method designed for production-grade accuracy retention. MTP Enabled model quantization
Quantization Details
| Parameter | Value |
|---|---|
| Method | AutoRound (W4A16) |
| Group Size | 32 |
| Symmetric | Yes |
| Iterations | 1200 |
| Calibration Samples | 512 |
| Sequence Length | 4096 |
| Torch Compile | Enabled |
Key Notes
- High accuracy configuration — 1200 iterations with 512 calibration samples targets production-grade quality with minimal degradation from the base model.
- W4A16 — Weights are quantized to 4-bit integers; activations remain in FP16 for inference stability.
- ~50% memory reduction compared to the FP16 base model, enabling deployment on consumer and mid-range GPUs.
- Vision Tower (
quant_nontext_module):False(Kept in BF16 to preserve visual reasoning and OCR precision) - Special Modules (
layer_config): Multi-Token Prediction (mtp,mtp.fc) kept in native bfloat16
MTP / Speculative Decoding
This model supports Multi-Token Prediction (MTP) for improved inference throughput using speculative decoding.
When serving with compatible backends (e.g., vLLM), enable MTP using:
--speculative_config '{"method":"mtp","num_speculative_tokens":1}'
Notes
num_speculative_tokens=1is a stable default for balancing speed and accuracy.- You can experiment with higher values for better throughput, depending on your hardware and latency requirements.
Usage
This model is compatible with transformers and backends that support AutoRound GPTQ-format weights (e.g., vLLM, SGLang, AutoGPTQ). For full model details, architecture, and capabilities, refer to the base model page.
🚀 Deploy on RunPod
One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.
🎁 Need GPU compute? Sign up via RunPod and get $5–$500 in free credits when you add your first $10.
PyTorch 2.14
| Template | CUDA Version | Docker Image | Template ID | Deploy |
|---|---|---|---|---|
| PyTorch 2.14 (CUDA 12.6) | 12.6 | vishva123/cuda-12.6-pytorch-2.14-runpod | d7lxsa4w9m |  |
| PyTorch 2.14 (CUDA 13.0) | 13.0 | vishva123/cuda-13.0-pytorch-2.14-runpod | yk0y6j6rpg | |
| PyTorch 2.14 (CUDA 13.2) | 13.2 | vishva123/cuda-13.2-pytorch-2.14-runpod | gsp4gwx0nw | |
PyTorch 2.13
| Template | CUDA Version | Docker Image | Template ID | Deploy |
|---|---|---|---|---|
| PyTorch 2.13 (CUDA 12.6) | 12.6 | vishva123/cuda-12.6-pytorch-2.13-runpod | gmlupxnxfk | |
| PyTorch 2.13 (CUDA 13.0) | 13.0 | vishva123/cuda-13.0-pytorch-2.13-runpod | y3j8xvk4f4 | |
| PyTorch 2.13 (CUDA 13.2) | 13.2 | vishva123/cuda-13.2-pytorch-2.13-runpod | vigpissn5w | |
PyTorch 2.12
| Template | CUDA Version | Docker Image | Template ID | Deploy |
|---|---|---|---|---|
| PyTorch 2.12 (CUDA 12.6) | 12.6 | vishva123/cuda-12.6-pytorch-2.12-runpod | ctmz86zmf0 | |
| PyTorch 2.12 (CUDA 13.0) | 13.0 | vishva123/cuda-13.0-pytorch-2.12-runpod | qjko5yiwzi | |
| PyTorch 2.12 (CUDA 13.2) | 13.2 | vishva123/cuda-13.2-pytorch-2.12-runpod | ifg6xmye0f | |