Load path (important)

Hub catalog is a compressed-tensors (pack-quantized) checkpoint. Prefer vLLM ≥ 0.21.

# Preferred
vllm serve 88plug/<ModelName> --trust-remote-code

Do not deploy via TGI “text-generation-inference” paths — that backend does not load our CT format and produces opaque worker/load errors.

Qwen2.5-Omni-7B-W4A16

INT4 post-training quantization of Qwen/Qwen2.5-Omni-7B — the 7B omni model with audio input, image input, and real-time speech output. ~3.5 GB on disk. Runs on any 8 GB GPU.

At a Glance

Memory Requirements

Quick Start

Tested with vLLM v0.21.0 (vllm/vllm-openai:v0.21.0-cu129-ubuntu2404). Weights are in compressed-tensors format — vLLM detects and loads quantization automatically. No --quantization flag needed.

vLLM — text output only

docker run --gpus device=0 -p 8080:8080 \
  vllm/vllm-openai:v0.21.0-cu129-ubuntu2404 vllm serve \
  88plug/Qwen2.5-Omni-7B-W4A16 \
  --kv-cache-dtype fp8 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90

Weights are in compressed-tensors format — no --quantization flag needed. Mainline vLLM returns text only. Audio input works; speech output does not.

Python client

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="unused")

response = client.chat.completions.create(
    model="88plug/Qwen2.5-Omni-7B-W4A16",
    messages=[{"role": "user", "content": "Explain the difference between W4A16 and W8A16 quantization."}],
    max_tokens=512,
)
print(response.choices[0].message.content)

vLLM-Omni — full audio output

vLLM-Omni v0.20.0 enables real-time speech output.

docker run --gpus device=0 -p 8080:8080 \
  vllm/vllm-omni:v0.20.0-cu129 vllm serve \
  88plug/Qwen2.5-Omni-7B-W4A16 \
  --kv-cache-dtype fp8 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90

SGLang — BF16 baseline

SGLang v0.5.8 does not support compressed-tensors natively. Run the BF16 base model for prefix-heavy and high-concurrency workloads.

docker run --gpus device=0 -p 30000:30000 \
  lmsysorg/sglang:v0.5.8-cu129 python -m sglang.launch_server \
  --model-path Qwen/Qwen2.5-Omni-7B \
  --tp 1 \
  --mem-fraction-static 0.85 \
  --port 30000

llama.cpp — audio/vision in, text out

Speech output is not in mainline llama.cpp (issue #21956). Convert from the BF16 base, not from the compressed-tensors checkpoint.

python convert_hf_to_gguf.py Qwen/Qwen2.5-Omni-7B \
  --outfile Qwen2.5-Omni-7B-BF16.gguf

llama-quantize Qwen2.5-Omni-7B-BF16.gguf Qwen2.5-Omni-7B-Q4_K_M.gguf Q4_K_M
llama-quantize --imatrix calibration_datav3.txt \
  Qwen2.5-Omni-7B-BF16.gguf Qwen2.5-Omni-7B-IQ4_XS.gguf IQ4_XS

llama-server \
  --model Qwen2.5-Omni-7B-Q4_K_M.gguf \
  --n-gpu-layers 999 \
  --ctx-size 32768 \
  --port 8081

Quantization Design

Recipe

AutoRound (blocked — see KNOWN-FAILURES) with scheme="W4A16", datafree RTN (QuantizationModifier), applied uniformly to all Linear targets in model.thinker. No mixed-precision or per-layer overrides.

What is quantized

All Linear modules in model.thinker (the LLM backbone — 28 transformer blocks) are quantized to W4A16 INT4 weights with BF16 activations and group size 128.

What stays BF16

Only the LLM backbone weights are quantized. The audio, vision, speech decoder, and vocoder components are untouched.

Calibration corpus

Quality Targets

W4A16 applies a wider KL tolerance than W8A16 by design — INT4 introduces more rounding error than INT8. The AutoRound (blocked — see KNOWN-FAILURES) optimizer and 200 calibration iterations are chosen to maximize recovery within this budget.

Competitor Comparables

Note: This is the only compressed-tensors W4A16 quant for Qwen2.5-Omni-7B at time of release, making it the only vLLM-native INT4 option for this model.

Benchmarks

Results from 88plug benchmark ladder — vLLM v0.21.0, lm-evaluation-harness. No fabricated numbers.

SGLang Note

SGLang v0.5.8 does not support compressed-tensors natively. Use the BF16 base model (Qwen/Qwen2.5-Omni-7B) with SGLang for prefix-heavy workloads. The W4A16 quant is vLLM-native.

Technical Details

Citation

@misc{qwen25omni,
  title  = {Qwen2.5-Omni Technical Report},
  author = {Qwen Team},
  year   = {2025},
  url    = {https://huggingface.co/Qwen/Qwen2.5-Omni-7B}
}

About

88plug AI Lab ships production-grade compressed-tensors quantizations for native vLLM v0.21.0+ deployment.

This release: Attention-gold tier — AutoRound W4A16-G32 on attention projections only (iters=200). MoE experts and multimodal towers stay BF16 by design.

Browse all releases → huggingface.co/88plug