Qwen3.8-27B-Pessoa-5090 (AR11)

Mixed-precision AutoRound INT quantization of Qwen/Qwen3.8-27B, saved in compressed-tensors (pack-quantized) format for direct use with vLLM.

This is a three-tier weight-only checkpoint tuned for the RTX 5090 / Blackwell SM120 memory envelope:

• MLP layers 0–31 — INT4 W4A16, group size 64, symmetric (heaviest compression on the early, less error-sensitive blocks).
• MLP layers 32–63 — INT5 W4A16, group size 64, symmetric (one extra bit on the late blocks where output distributions are more sensitive).
• All attention projections (self-attn Q/K/V/O + linear-attn qkv/z/out) — INT6 W6A16, group size 128, symmetric (attention kept higher-precision to preserve routing/recall fidelity).
• Activations are not quantized (weight-only): weights are dequantized to BF16 at the kernel boundary, so there is zero activation quantization error.
• Visual encoder, MTP head, lm_head, embeddings, and linear-attention recurrent projections (in_proj_a, in_proj_b, norm, conv1d) are kept in BF16.

The 32/32 INT4-vs-INT5 MLP split plus INT6 attention lands at 22.59 GB — small enough to fit comfortably on a single RTX 5090 (32 GB) with room for KV cache and MTP speculative decoding — while holding KLD to 0.00397 nats (top-1 agreement 97.64%) on the 278,392-position fidelity suite.

Model Details

Qwen3.8-27B (qwen3_5 / Qwen3_5ForConditionalGeneration):

• Type: Vision-language model (hybrid linear + full attention language model + vision encoder)
• Parameters: 27.79B total
• Layers: 64 — 48 linear-attention (GatedDeltaNet) + 16 full-attention (every 4th layer: 0, 4, 8, …, 60)
• Hidden size: 5,120
• Intermediate size: 17,408
• Attention heads: 24 × 256 dim (full-attn); 16 key × 128 dim / 128 value head (linear-attn)
• Vocabulary: 248,320
• Activation: SiLU (SwiGLU MLP, output_gate_type=swish)
• Context: 262,144 tokens (max_position_embeddings)
• Vision encoder: 27-layer ViT, hidden size 1,152, patch size 16, 16 heads
• MTP: 1 layer, num_speculative_tokens=3, shared embeddings

Each linear-attention layer has in_proj_qkv, in_proj_z, out_proj plus recurrent state paths in_proj_a / in_proj_b, a norm, and a conv1d (kernel dim 4). Each full-attention layer has standard q_proj / k_proj / v_proj / o_proj (4 KV heads, partial rotary 0.25). All layers share gate_proj / up_proj / down_proj MLP blocks. The vision tower (model.visual.*) processes image/video patches and injects embeddings into the language model via image/video token boundaries.

Quantization Configuration

Precision Assignment

Weight-only: input_activations: null, output_activations: null for all groups. Weights are dequantized to BF16 at the kernel boundary — no activation quantization error. KV cache is left in BF16 (no kv_cache_scheme).

Modules Kept in BF16

Calibration

• Tool: AutoRound 0.14.2 (MLLM block-walker path)
• Iters: 200
• Source: malaiwah/qwen38-27b-fidelity-suite-v3 tokens (181 contexts × 2048 tokens)
• Samples: 181, max sequence length 2048
• Format: Decoded token IDs from the fidelity suite, split 2/3 user / 1/3 assistant
• Scheme base: W4A16, overridden per-module by an explicit layer_config (exact module names — AutoRound 0.14.2 does not expand regex keys for the MLLM path)
• Seed: 0

Build command

python scripts/quant_pessoa_autoround_v7.py \
  --model /models/Qwen3.8-27B \
  --output /models/Qwen3.8-27B-Pessoa-AR11-w6-late5-32 \
  --iters 200 --calib-samples 181 --calib-max-len 2048 \
  --lat-bits 6 --sat-bits 6 \
  --late-mlp-bits 5 --late-mlp-start 32 \
  --base-mtp-dir /models/Qwen3.8-27B

Fidelity Evaluation (KLD)

Distribution-fidelity measured against the BF16 base on the qwen38-27b-fidelity-suite-v3 capture-replay harness: 136 contexts, 278,392 scored positions, comparing the full next-token softmax distribution (KL divergence in nats, float64 accumulation, two-pass) using the shared BF16 lm_head.

Headline Metrics

Bootstrap CI (mean KLD, 10,000 resamples, 41 clusters)

KLD by Stratum

As is typical for weight-only quantization of this family, the literary stratum (Project Gutenberg prose) carries the highest mean KLD, while scientific and multilingual contexts are the closest to the base distribution.

Worst Contexts

The worst individual token (max KLD 1.50 nat) occurs in a Moby-Dick passage; even there the top-1 agreement remains above 97%.

Comparator

• Vocab chunk: 24,832 · Device: CUDA · Accumulation: float64 · Two-pass: true

AutoRound Ladder (Pessoa series)

AR11 sits in a sweep of AutoRound mixed-precision variants, all with INT6 attention (lat + sat) and a progressively later INT4→INT5 MLP boundary:

KLD falls monotonically as the INT4→INT5 boundary moves earlier (more layers at INT5). AR11's 32/32 split is the knee of the curve: each step beyond it yields diminishing fidelity gains for the added size, while each step before it costs noticeably more KLD.

Usage

# vLLM serve (RTX 5090 / Blackwell SM120 + FlashInfer)
vllm serve /models/Qwen3.8-27B-Pessoa-AR11-w6-late5-32 \
  --served-model-name qwen38-27b-pessoa-5090 \
  --max-model-len 131072 \
  --max-num-seqs 32 \
  --quantization compressed-tensors \
  --trust-remote-code \
  --attention-backend flashinfer \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

The endpoint is OpenAI-compatible. MTP speculative decoding is enabled (num_speculative_tokens=3), providing a decode speedup when acceptance is high.

Hardware & Runtime

Limitations

• No task-accuracy benchmarks yet. This model has a KLD fidelity report but has not been run through the GSM8K / MMLU-Pro / GPQA-Diamond accuracy suite. KLD is a strong proxy for distribution closeness but is not a direct task-score measurement. Validate on your own workload before production use.
• Quantization is lossy. Weight-only INT4/INT5/INT6 eliminates activation error but retains weight rounding error. The literary stratum shows the highest divergence.
• Blackwell-optimized. The INT pack-quantized MMA paths are exercised on SM120 (Blackwell). Validate on your hardware (Ada/Hopper SM89/SM90 also support the relevant integer paths).
• AutoRound 0.14.2 MLLM path quirk. Regex layer_config keys are not expanded for the MLLM block-walker; this checkpoint uses explicit per-module keys (400 entries) and a post-build config guard verifying exported group bits match the recipe.
• RTX 5090 naming. The "5090" in the model name denotes the target memory envelope (single 32 GB card). The checkpoint is hardware-agnostic compressed-tensors and runs on any vLLM-supported GPU with sufficient VRAM.

Acknowledgements

Standing on the shoulders of giants. This work would not exist without the rtx6kpro community and the broader local-inference-lab Discord — the open protocol, tooling, and baseline measurements that made systematic quantization comparison possible.

The fidelity suite (malaiwah/qwen38-27b-fidelity-suite-v3) and the KLD capture-replay harness provided the distribution-level validation. AutoRound (Intel) produced the optimized rounding.