Qwen3.8-Flash-Next-MXFP4-Mixed-CT-RTN-AutoRound

Model Details

This is a MXFP4-Mixed quantization of Qwen/Qwen3.8-Flash-Next, generated by AutoRound. Please follow the license of the original model.

Quantization Details

• Scheme: MXFP4-Mixed
• Method: AutoRound
• Base model: Qwen/Qwen3.8-Flash-Next

Evaluation

Reproduce & Environment

Quantized with:

auto-round --model_name /models/lkk/cri_models/Qwen3.8-Flash-Next --scheme MXFP8 --model_free --ignore_layers visual,lm_head,embed_tokens,mlp.gate,mlp.shared_expert_gate,in_proj_a,in_proj_b,block_inject_weight,ple,mtp,hyper_connection --layer_config '{"mlp.experts":{"bits":4,"data_type":"mx_fp"}}' --device_map auto --format llm_compressor --output_dir /models/auto-quant/results/qwen38_flash_next/mxfp4_mixed_fp8kv_tp1/quantized

Evaluated with:

lm_eval --model vllm --model_args pretrained=/models/auto-quant/results/qwen38_flash_next/mxfp4_mixed_fp8kv_tp1/quantized,tensor_parallel_size=1,max_model_len=8192,gpu_memory_utilization=0.85,dtype=bfloat16,trust_remote_code=True,add_bos_token=True,enable_prefix_caching=False,max_gen_toks=2048,max_num_batched_tokens=16384,max_num_seqs=64,language_model_only=True,reasoning_parser=qwen3,enable_thinking=False,kv_cache_dtype=fp8,safetensors_load_strategy=prefetch --tasks gsm8k --batch_size 64 --output_path /models/auto-quant/results/qwen38_flash_next/mxfp4_mixed_fp8kv_tp1/eval/gsm8k --log_samples --seed 42 --num_fewshot 5 --apply_chat_template --fewshot_as_multiturn
lm_eval --model vllm --model_args pretrained=/models/auto-quant/results/qwen38_flash_next/mxfp4_mixed_fp8kv_tp1/quantized,tensor_parallel_size=1,max_model_len=8192,gpu_memory_utilization=0.85,dtype=bfloat16,trust_remote_code=True,add_bos_token=True,enable_prefix_caching=False,max_gen_toks=2048,max_num_batched_tokens=16384,max_num_seqs=64,language_model_only=True,reasoning_parser=qwen3,enable_thinking=False,kv_cache_dtype=fp8,safetensors_load_strategy=prefetch --tasks piqa,mmlu,hellaswag --batch_size 32 --output_path /models/auto-quant/results/qwen38_flash_next/mxfp4_mixed_fp8kv_tp1/eval/piqa_mmlu_hellaswag --log_samples --seed 42

Inference: load this repo with a vllm build that supports the MXFP4-Mixed format (day-0 architectures may need the model card's custom image/branch — check the quantize command above for the environment used).

Produced with autoquant-agent — agent-driven quantize + evaluate + self-heal.

Serving notes (measured on this artifact — read before benchmarking it)

• FP8 KV cache is a runtime flag, not part of the weights. quantization_config.kv_cache_scheme is null here by design: Qwen4Exp's QSA path raises on any KV-quantization written into the checkpoint. Turn it on at serve/eval time with --kv-cache-dtype fp8 (vLLM engine arg; lm_eval passes it through --model_args kv_cache_dtype=fp8). Paired A/B on this checkpoint (n=300, same seed/build, only the flag differing): every delta < 1σ (piqa acc_norm +0.007, hellaswag acc_norm −0.007), so FP8 KV costs nothing measurable at 8k context.
• tensor_parallel_size=1 is required for correct numerics with this moe_intermediate_size=640 checkpoint: the CUTLASS MXFP4 MoE needs moe_intermediate/TP % 128 == 0. At TP=2 main raises (shape '[2560, 10]' is invalid for input of size 30720), and reshaping the swizzled scales to the padded dims "fixes" the crash while silently producing garbage (piqa 0.51 ≈ chance, degenerate repetition). TP=1 additionally works here only because recent vLLM runs the 102 GB n-gram PLE table from pinned host memory (EngramConfig(cpu_offload=True)); on builds where that table is GPU-resident, TP=1 OOMs and TP≥2 is mandatory — i.e. this artifact has no verified TP>1 path.
• vLLM must be ≥ the commit carrying PR #55557 for FP8 KV (fp8_e4m3 added to QSA's supported_kv_cache_dtypes). Not in v0.29.0 / v0.30.0 — verified 2026-09-22. Measured build: 0.29.1rc1.dev528+g9815732c1.
• compressed-tensors checkpoints of this arch need a patch on that same build: PR #54371 turned the PLE-embedding method selection into a whitelist that omits CompressedTensorsConfig, so model construction raises Qwen4Exp PLE embedding does not support quantization config CompressedTensorsConfig before any weight is read.
• Other eval口径: language_model_only=True, dtype=bfloat16, enable_thinking=False, reasoning_parser=qwen3, seed=42, no prefix caching, no expert parallel, no spec decode, PP=1.

Evaluation notes

• Table above uses lm_eval's acc,none for multiple-choice tasks (vLLM/aqa leaderboard convention). Length-normalized numbers for the same run: piqa 0.8341, hellaswag 0.8704 (hellaswag's un-normalized acc sits ~19pp lower by task design — compare conventions before reading a drop into these numbers). Mean over the 4 tasks: 0.8326 (acc) / 0.8830 (acc_norm), against same-box archived references bf16 0.8825 and W4A16-RTN 0.8819.
• Long context, same checkpoint + FP8 KV, 100 examples each at 131 072 tokens (lm_eval niah_multiquery, ruler_qa_squad): 0.8300 and 0.8083 Full logs in the aqa results dataset: datasets/lvkaokao/lb_local → results/Qwen/Qwen3.8-Flash-Next-MXFP4-Mixed-CT-RTN-AutoRound/run_*（量化+短上下文 run 与 128k 长上下文 run 各一条，含 run_manifest.json、lm_eval results/samples 与逐命令日志）。