Qwen3.5-9B-MTP-Mixed-8bit

Mixed-precision 8-bit quantization of Qwen3.5-9B — a 9B multimodal vision-language model with hybrid attention and multi-token prediction, optimized for efficient local inference with minimal quality loss.

Quick Stats

Quantization Strategy

This model uses surgical mixed-precision quantization: only the SwiGLU MLP projection matrices (gate_proj, up_proj, down_proj) in the language model transformer layers are quantized to 8-bit. All other components remain in BF16:

• Quantized (8-bit): LLM MLP layers — 96 tensors across 32 layers (~66% of model mass)
• Protected (BF16): Linear attention (24 layers), full attention (8 layers), vision tower (333 tensors), MTP head (15 tensors), embeddings, LM head, all normalization layers, linear attention SSM parameters (A_log, dt_bias, conv1d)

Rationale: MLP layers are "knowledge lookup" / feature transformation layers that are extremely quantization-tolerant. Attention mechanisms carry reasoning and context-window recall. The vision tower and MTP head are kept in full precision to preserve multimodal understanding and multi-token prediction quality.

Model Architecture

Language Model

• 32 dense layers alternating [linear_attention × 3, full_attention × 1] × 8
• Hybrid attention: SSM-style linear attention (24/32 layers, 75%) + full MHA with GQA 4:1 (8/32 layers, 25%)
• Hidden size: 4096, Intermediate: 12288
• Heads: 16Q / 4KV (GQA), Head dim: 256
• Linear attention: key dim 128, value dim 128, conv kernel 4
• RoPE: MRoPE interleaved, section [11, 11, 10], theta=10,000,000, partial rotary 0.25

Vision Tower (BF16)

• 27-block ViT, hidden 1152, intermediate 4304, 16 heads
• Patch size: 16×16, Spatial merge: 2×2
• Output projection: 1152 → 4096 (via merger with norm + FC layers)
• Supports image and video input (temporal patch size: 2)

MTP Head (BF16)

• 1-layer multi-token prediction with dedicated full attention
• Pre-FC norm (hidden + embedding), fusion linear, decoder layer (attention + MLP + norms), final norm
• Shares embeddings with main model (mtp_use_dedicated_embeddings: false)

Tokenizer

• Vocab size: 248,320
• Special tokens: image (248056), video (248057), vision start (248053), vision end (248054)

Quantization Details

Script

Quantized using mixed_quantize_qwen3_5_9b_v3.py — an MTP-aware quantization pipeline that:

1. Monkey-patches mlx_vlm to preserve MTP keys through Model.sanitize() (stock mlx_vlm strips mtp.* keys)
2. Patches TextConfig.from_dict to retain mtp_num_hidden_layers
3. Patches LanguageModel.__init__ to attach MTPModule when config declares MTP layers
4. Quantizes MLP-only via nn.quantize() with surgical class predicate
5. Writes MLX format metadata ({"format": "mlx"}) to safetensors shards

Tensor Breakdown

MTP Key Preservation

The 15 MTP tensors are stored with mtp. prefix in the weight index (HF-style naming). When loading with patched mlx_vlm, these are renamed to language_model.mtp.* during sanitization. The keys preserved:

mtp.fc.weight
mtp.norm.weight
mtp.pre_fc_norm_hidden.weight
mtp.pre_fc_norm_embedding.weight
mtp.layers.0.{self_attn.{q,k,v,o,k_norm,q_norm}_proj.weight, mlp.{gate,up,down}_proj.weight, {input,post_attention}_layernorm.weight}

Usage

oMLX (Recommended)

Loads natively with MTP and vision support. The MLX format metadata in safetensors ensures sanitize() is skipped on reload, preserving MTP keys.

MLX / mlx-vlm

from mlx_vlm.utils import load

model, processor = load("your-username/Qwen3.5-9B-MTP-Mixed-8bit", lazy=True)
# Model loads in ~14 GB vs ~18 GB for BF16

Note: If using stock mlx_vlm without the MTP patches from the quantization script, MTP keys may be stripped during loading. Use oMLX or apply the patches.

Files

Limitations

• Mixed quantization may show minor quality degradation on edge cases compared to BF16
• This is a direct quantization of the base model — no fine-tuning was applied after quantization
• MTP head quality reflects the base Qwen3.5-9B model (1 prediction layer)
• The linear attention SSM parameters (A_log, dt_bias, conv1d) are kept in BF16/FP32 as specified by the architecture; these are sensitive to precision and were explicitly protected

License

Apache 2.0 — same as Qwen/Qwen3.5-9B