Qwen3.8-27B-GPTQ-INT4-FP8KV

A 4-bit weight-only GPTQ quantization of Qwen/Qwen3.8-27B with an fp8-e4m3 KV cache calibrated alongside the weights. INT4 weights shrink the footprint ≈2.8× and fp8 KV halves the cache, so the model serves its native 262,144-token context on a single 48 GB GPU. fp8 KV also makes YaRN extension to 524,288 tokens fit the same card, and up to 1M tokens on 96 GB (retrieval beyond 262k unvalidated).

Fits a single 48 GB GPU

TL;DR

Evaluation

Identical harness and hardware for base and quantized: lm-evaluation-harness with the vLLM 0.27.1 backend on a single NVIDIA RTX PRO 6000 (Blackwell, 96 GB), greedy decoding, seed 1234. The base runs with bf16 KV and the quantized model with fp8-e4m3 KV, each at its recommended serving configuration, so every Δ measures the full int4 + fp8-KV stack rather than weights alone. Generative tasks run with the model's native chat template, thinking enabled, max_gen_toks=8192; loglikelihood tasks run untemplated. minerva_math500 reports math_verify, since strict exact_match measures answer format, not content, on thinking-model outputs.

All deltas are within sampling noise for their sample sizes.

Long-context retrieval (needle-in-a-haystack, thinking, greedy)

The BF16 base scores identically (9/9), showing no retrieval degradation through the int4 + fp8-KV stack at the context length it was calibrated at.

Limitations

• Generation-task scores can vary by a few points across GPU vendors even for identical configs; base and quantized were measured on the same machine to keep every Δ above a within-config comparison.

What was quantized (and what was protected)

Unlike the common practice of keeping DeltaNet projections in BF16, they are quantized here. The eval suite therefore deliberately includes vision and long-context retrieval to cover that risk.

Serving

The checkpoint carries its own calibrated k/v scales, and --kv-cache-dtype fp8_e4m3 picks them up automatically.

What fits:

• 48 GB + fp8 KV: native 262k context, comfortable (≈33 GB used)
• 48 GB + bf16 KV: 262k fits but tight (≈42 GB); drop --kv-cache-dtype
• 48 GB + fp8 KV + YaRN (factor: 2.0): 524k context
• 96 GB + fp8 KV + YaRN (factor: 4.0): up to 1M context

Single 48 GB GPU: 262k, fp8 KV (the default)

vllm serve abhishekchohan/Qwen3.8-27B-GPTQ-INT4-FP8KV \
  --kv-cache-dtype fp8_e4m3 \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 4 \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

fp8 KV is the default on 48 GB; bf16 KV is a 96 GB option. If you OOM at full context on 48 GB, drop --speculative-config first.

Single 96 GB GPU: 262k with concurrency headroom

vllm serve abhishekchohan/Qwen3.8-27B-GPTQ-INT4-FP8KV \
  --kv-cache-dtype fp8_e4m3 \
  --max-model-len 262144 \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

On a 96 GB card (RTX PRO 6000 Blackwell) the fp8 KV pool holds ≈2M tokens (≈8× the max context), so raise --max-num-seqs for concurrency. At bf16 KV the full 262k context needs ≈17 GB of KV (≈65 KB/token, since only the 16 global-attention layers accumulate KV); drop --kv-cache-dtype if you prefer bf16 KV.

Host notes. On CUDA-12.x-driver hosts install the +cu129 vLLM wheel (PyPI wheels are cu13-only from 0.26) and run with --attention-backend TRITON_ATTN + VLLM_USE_FLASHINFER_SAMPLER=0; flashinfer JIT additionally needs a CUDA ≥ 12.9 toolkit (the pip nvidia-cuda-*-cu13 packages supply it). On CUDA-13 hosts stock vLLM works as-is.

Beyond 262k (YaRN). The base architecture extends via YaRN, and fp8 KV is what makes the long end practical on a single card. For 524,288 tokens on one 48 GB GPU use factor: 2.0 (for 1M on 96 GB use factor: 4.0):

VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve abhishekchohan/Qwen3.8-27B-GPTQ-INT4-FP8KV \
  --kv-cache-dtype fp8_e4m3 \
  --hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 2.0, "original_max_position_embeddings": 262144}}}' \
  --max-model-len 524288

Static YaRN can mildly hurt short-text quality, so apply the override only when you need the extra context. This checkpoint was calibrated and validated at the native 262,144 tokens, so re-check retrieval at your target length before relying on anything beyond that.

Quantization recipe

GPTQ with weight-only INT4: asymmetric (zero-point, stored as int8), group_size=32, imatrix-mse weight observer (importance statistics collected by forward hooks over the calibration rows), actorder=static, block size 128, dampening_frac: 0.01, applied to every Linear layer not protected in the table above. The fully resolved recipe ships as recipe.yaml in this repo.

Long-sequence calibration. Conversations from five Nemotron Post-Training v3 SFT datasets (instruction-following chat, math, science, agentic code, multilingual) were pooled (≈37M tokens), shuffled (seed 42), and packed EOS-separated into 8 fixed 262,144-token rows, calibrating at the exact context length the model is served at. Calibration texts are rendered with thinking enabled (reasoning traces preserved), matching the deployment regime.

KV cache. The same oneshot pass observes k/v activations of the 16 softmax-attention layers (the 48 DeltaNet layers carry recurrent state, not a KV cache) and bakes the resulting per-tensor scales into the checkpoint (range across layers: 0.016–0.203).