Qwen3.8-2.4T-A95B
A complete converted checkpoint of Qwen/Qwen3.8-2.4T-A95B.
Tested baseline launch
This is a single-active-request profile. Install the compatible Furnace runtime first, then run:
export ROCR_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export SGLANG_USE_AITER=1
export SGLANG_MOE_PADDING=1
export ROCM_QUICK_REDUCE_QUANTIZATION=NONE
export OMP_NUM_THREADS=8
export MAX_JOBS=8
sglang serve \
--model-path m8than/Qwen38-2.4T-GG \
--served-model-name Qwen/Qwen3.8-2.4T-A95B \
--dtype bfloat16 --tp-size 8 --ep-size 1 --dcp-size 1 \
--attention-backend triton \
--linear-attn-prefill-backend triton \
--linear-attn-decode-backend triton \
--linear-attn-verify-backend triton \
--mamba-ssm-dtype float32 \
--disable-shared-experts-fusion \
--moe-runner-backend aiter --moe-a2a-backend none \
--mem-fraction-static 0.90 \
--context-length 262144 --max-total-tokens 262144 \
--max-running-requests 1 --max-mamba-cache-size 16 \
--cuda-graph-max-bs-decode 1 \
--chunked-prefill-size 8192 --max-prefill-tokens 8192 \
--load-format safetensors \
--model-loader-extra-config '{"enable_multithread_load":true,"num_threads":4}' \
--reasoning-parser qwen3 --tool-call-parser qwen3_coder \
--watchdog-timeout 1200 --host 127.0.0.1 --port 8000
On shared/network storage, --weight-loader-prefetch-checkpoints --weight-loader-prefetch-num-threads 1 can improve startup loading. These flags do not change inference math.
Measured serving throughput
Measured on eight MI325X 256 GiB GPUs, PyTorch 2.11.0+rocm7.2, HIP 7.2.26015, and Triton 3.7.0. One active request, warmed server and prefix cache, thinking enabled with reasoning effort low. Counts include reasoning tokens. Tokenization precedes timing.
Coding is the median of three runs with 84 input tokens and 2,048 generated tokens. Prose is one run with 77 input tokens and 1,024 generated tokens. Temperature-1 sampling uses top-p 0.95 and top-k 20.
| Configuration | Temperature | Coding decode tok/s, median | Prose decode tok/s |
|---|---|---|---|
| No speculation | 1.0 | 45.86 | 46.07 |
| DSpark, block 7 | 1.0 | 129.19 | 94.12 |
| DSpark, block 7 | 0.0 | 138.25 | 94.70 |
DSpark's temperature-1 coding median was 128.09 tok/s end-to-end, and its greedy coding median was 136.98 tok/s end-to-end. These are workload-specific generation measurements, not full-context prefill, multi-user throughput, or a model-quality benchmark. The length-limited coding responses are not asserted to be complete modules.
The speculative runs used RadixArk/Qwen3.8-2.4T-A95B-DSpark, revision faaee89a1b8483d4031375b312d7d3eafae36d57, with SGLANG_RAGGED_VERIFY_MODE=static, SGLANG_DSPARK_FOLDED_SAMPLING=0, and these additions to the baseline command:
--speculative-algorithm DSPARK
--speculative-draft-model-path RadixArk/Qwen3.8-2.4T-A95B-DSpark
--speculative-draft-model-revision faaee89a1b8483d4031375b312d7d3eafae36d57
--speculative-dspark-block-size 7
--speculative-draft-attention-backend triton
--speculative-draft-kv-cache-dtype bfloat16
Validation and limitations
The complete export's routed-weight relative L2 error versus BF16 is 12.71%. On one real expert with synthetic inputs, the compiled CK output differed from a software W4A8 reference by at most 0.0084% relative L2. Intentional quantization differed from the original BF16 expert output by 18–19.2% on those inputs. Weight-space and single-expert errors are not accuracy scores.
The loaded full model passed arithmetic generation checks, an OpenAI chat check, and eleven executed cases for a generated Python prime-checking function. Three greedy smoke outputs matched with and without DSpark. Broad accuracy and perplexity evaluation has not been performed; BF16-equivalent quality and arbitrary-prompt bitwise determinism are not claimed.
Target weights occupied approximately 150.27 GiB per GPU, excluding draft weights and runtime workspaces; the configured target KV cache occupied 5.75 GiB per GPU. Four 256 GiB GPUs cannot hold this checkpoint fully GPU-resident.
License
This derivative retains the upstream Qwen3.8-Max License, provided in LICENSE. Its commercial-use conditions continue to apply. Consult the upstream model and license for the base model's terms. The upstream BF16 benchmark claims are not measurements of this converted checkpoint.