Qwen3.8-Flash-Next-DERISKED-W4A16-AWQ
A W4A16 (AWQ) quantization of Blackfrost-AI/Qwen3.8-Flash-Next-DERISKED-BF16,
an abliterated ("derisked") variant of Qwen's Qwen3.8-Flash-Next.
Multimodal (image-text-to-text), MTP speculative-decode head and PLE n-gram table preserved.
Built to run the 80B-class Flash-Next MoE on 2× AMD MI210 (gfx90a) at long context.
What this is
- Format:
compressed-tensors, W4A16 — 4-bit weights, group size 128, symmetric, no zero-point (loads on vLLM'sTritonW4A16+fused_moe_kernel_gptq_awqfast paths). - Method: AWQ (activation-aware). Smoothing is restricted to the mathematically-exact
up_proj → down_projpairs. The standardnorm → q/k/vandnorm → gate/upsmoothings are deliberately excluded: this architecture uses hyper-connections with a data-dependent (sigmoid) gate, so folding a smoothing scale into the norm would change the gate non-linearly and corrupt the weights. - Quantized: routed experts (gate/up/down) and full-attention
q/k/v/o. - Kept in bf16 (matching the reference recipe): the GDN linear-attention layers (recurrent state — error compounds along the sequence), the shared expert, the QSA indexer, the router gates, hyper-connections, norms,
embed_tokens,lm_head, the MTP speculative-decode head, and the PLE n-gram embedding table.
Performance (measured, 2× MI210, vLLM, TP=2, 256K context)
| Metric | Value |
|---|---|
| Prefill | ~2,200 tok/s (rises with prompt length) |
Decode (MTP spec decode, num_speculative_tokens=3) | ~40 tok/s |
| Decode (no spec decode) | ~10 tok/s |
| Context | 262,144 (256K) |
Decode is bandwidth-bound: the ~100 GB PLE table is CPU-offloaded and streamed over PCIe, so single-token decode is inherently slow; the MTP head (spec decode) recovers most of it. Prefill is compute-bound and rides the tuned int4 MoE kernels.
Note on
num_speculative_tokens: 3 is the practical ceiling on this stack — n≥5 trips a QSA-indexer constraint (ring capacity 12 must divide the attention block size).
Serving (vLLM, ROCm / MI210)
vllm serve Qwen3.8-Flash-Next-DERISKED-W4A16-AWQ \
--tensor-parallel-size 2 --enable-expert-parallel --dtype bfloat16 \
--gpu-memory-utilization 0.95 --max-model-len 262144 --max-num-seqs 4 \
--block-size 64 --cpu-offload-params ngram_embedding --cpu-offload-gb 60 \
--moe-backend triton --mamba-cache-dtype bfloat16 \
--speculative-config '{"method":"qwen4_exp_mtp","model":"Qwen3.8-Flash-Next-DERISKED-W4A16-AWQ","num_speculative_tokens":3}' \
--disable-custom-all-reduce --trust-remote-code
Provenance & credits
- Base model: Qwen/Qwen3.8-Flash-Next — © Qwen.
- Abliteration: Blackfrost-AI/Qwen3.8-Flash-Next-DERISKED-BF16.
- Quantization: this repo (W4A16 / AWQ via llm-compressor).
License
Qwen Community License 1.0 (qwen-community-1.0), inherited from the base model. See the included LICENSE and the upstream Qwen3.8-Flash-Next LICENSE. This quantization is a derivative work and is distributed under the same terms.
Disclaimer
This is an abliterated / uncensored model: safety refusal behavior has been removed. You are responsible for how you use it and for compliance with the base-model license and applicable law.