Qwen3.8-Max DSpark speculator
A DSpark speculator for RadixArk/Qwen3.8-2.4T-A95B-NVFP4. DSpark extends the DFlash parallel-draft backbone with a Markov logit-bias head and a per-position confidence head. Trained with SpecForge online distillation from hidden states captured off a live SGLang target engine.
Model specifications
- Base model:
RadixArk/Qwen3.8-2.4T-A95B-NVFP4(92-layer hybrid-attention MoE, hidden size 8192) - Format: single-file BF16 safetensors, 3,298,141,697 parameters
- Draft: 5 full-attention Qwen3-style GQA layers, hidden size 8192, 64 query heads / 8 KV heads at head_dim 128,
block_size=7 - Verification width: 1 current token + 7 draft tokens
- Auxiliary target layers:
[3, 19, 35, 55, 71, 87], plus the final post-norm hidden state - Positional scheme: YaRN factor 32 with original length 8,192 — 262,144 addressable positions, the target's native context limit
- Target weights: embedding and unembedding weights are not included
Acceptance length
Accept length is the mean number of accepted tokens per verify step including the bonus token (SGLang spec_accept_length), averaged over requests. Setting: SGLang DSPARK serving, TP8 on one 8xB300 node, NVFP4 target, BF16 draft (--speculative-draft-model-quantization unquant), block size 7; sampling temperature 0.6, top_k 20, top_p 0.95, thinking enabled; max_new_tokens 2048; 128 prompts per workload (full set where the dataset is smaller). The measurement loop follows z-lab/dflash benchmark.py.
| Workload | Accept length | Prompts |
|---|---|---|
| humaneval | 5.30 | 128 |
| gsm8k | 4.89 | 128 |
| math500 | 4.85 | 128 |
| livecodebench | 4.72 | 128 |
| mbpp | 4.52 | 128 |
| aime25 | 4.43 | 30 |
| lbpp | 4.40 | 128 |
| aime26 | 4.35 | 30 |
| mtbench | 3.43 | 80 |
| arena_hard_v2 | 3.22 | 128 |
| alpaca | 3.05 | 128 |
Serving with SGLang
sglang serve \
--trust-remote-code \
--model-path RadixArk/Qwen3.8-2.4T-A95B-NVFP4 \
--tp-size 8 \
--speculative-algorithm DSPARK \
--speculative-draft-model-path RadixArk/Qwen3.8-2.4T-A95B-DSpark \
--speculative-dspark-block-size 7 \
--speculative-draft-attention-backend trtllm_mha