Qwen3.5-4B-AWQ

AWQ INT4 quantization of Qwen/Qwen3.5-4B using llm-compressor and 512 OpenPlatypus calibration samples.

2.56× smaller on disk and ~61.7% lower VRAM usage while maintaining strong benchmark performance.

Model compression

Benchmarks

Note: these are the quantized model's standalone scores from EleutherAI lm-evaluation-harness, default settings, 0-shot. HellaSwag and ARC-Challenge use acc_norm; PIQA, Winogrande, and ARC-Easy use acc, matching each task's harness default. A matched BF16-vs-INT4 delta on identical hardware and settings has not yet been run for this model; treat the scores below as standalone results rather than a verified quantization delta.

Average score: 68.67%

Quantization recipe

Calibration used real instruction-following data from OpenPlatypus rather than data-free quantization techniques.

Usage

With transformers

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

MODEL_ID = "kumar2235/Qwen3.5-4B-AWQ"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

prompt = "Explain machine learning in one paragraph."

inputs = tokenizer(
    prompt,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=256
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

With vLLM

from vllm import LLM, SamplingParams

llm = LLM(
    model="kumar2235/Qwen3.5-4B-AWQ"
)

outputs = llm.generate(
    ["Explain machine learning in one paragraph."],
    SamplingParams(
        temperature=0.7,
        max_tokens=256
    )
)

print(outputs[0].outputs[0].text)

Sample output

Prompt:

Explain machine learning in one paragraph.

Response:

Machine learning is a branch of artificial intelligence that enables computers to learn patterns from data and improve their performance on tasks without being explicitly programmed for every situation. By analyzing large amounts of information, machine learning models can make predictions, classify data, recognize patterns, and support decision-making. It powers applications such as recommendation systems, image recognition, language translation, fraud detection, and autonomous systems.

Hardware

• Weights: ~3.13 GB on disk, ~3.06 GB VRAM at load
• Single-GPU friendly: comfortably fits on 8 GB+ consumer cards for local inference and edge deployment

Limitations

• Benchmarks above are standalone scores for the quantized model; they have not yet been diffed against a BF16 run under identical harness settings, so the true accuracy delta from quantization is not yet confirmed
• Calibration set was OpenPlatypus (English-leaning instruction data) — heavy non-English or domain-specific workloads may benefit from re-quantizing on a matching corpus
• Max sequence length used during calibration was 1024 tokens; behavior at much longer contexts has not been separately validated

License

Inherits the license of the base model. See the Qwen/Qwen3.5-4B model page for terms.

Citation

Base model

@misc{qwen3.5-4b,
    title  = {{Qwen3.5-4B}},
    author = {{Qwen Team}},
    year   = {2025},
    url    = {https://huggingface.co/Qwen/Qwen3.5-4B}
}

Quantization method

@article{lin2023awq,
    title   = {{AWQ}: Activation-aware Weight Quantization for LLM Compression and Acceleration},
    author  = {Lin, Ji and Tang, Jiaming and Tang, Haotian and Yang, Shang and Chen, Wei-Ming and Wang, Wei-Chen and Xiao, Guangxuan and Dang, Xingyu and Gan, Chuang and Han, Song},
    journal = {arXiv preprint arXiv:2306.00978},
    year    = {2023}
}

Storage Format

This model uses the compressed-tensors format.

Hugging Face may display BF16/I32/I64 tensor types because compressed AWQ models store quantization metadata, scales, and packed weights separately. The model loads and runs as a compressed AWQ INT4 model through Transformers and llm-compressor.