GLM-5.3-W4AFP8
W4AFP8 quantization of zai-org/GLM-5.3, quantized from the BF16 master: MoE expert weights in INT4 (group-128, AWQ-calibrated), activations and non-expert layers in FP8, served by SGLang's native CUTLASS W4A8 grouped GEMM on Hopper (SM90). Roughly 2x the KV-cache capacity of the FP8 checkpoint at identical serving throughput, with EAGLE/MTP speculative decoding intact.
Also mirrored on ModelScope (魔搭) for faster downloads in China.
Successor to PhalaCloud/GLM-5.2-W4AFP8; GLM-5.3 is architecturally identical to GLM-5.2 (verified tensor-by-tensor), so the same serving configuration applies.
What is different from the 5.2 checkpoint
- Quantized from the BF16 master (
zai-org/GLM-5.3-BF16), not the FP8 release, avoiding double quantization of expert weights. - AWQ calibration on coding-agent traces (SALT-NLP/SWE-chat, 128 windows x 2048 tokens, session-disjoint split), matching the production agentic workload, instead of generic web text.
- The MTP/nextn draft layer (layer 78) ships quantized; EAGLE speculative decoding works out of the box.
Evaluation
Measured on this checkpoint served with the SGLang configuration below (official request parameters, reasoning at the chat-template default).
| Benchmark | GLM-5.3-W4AFP8 | GLM-5.3 reference |
|---|---|---|
| GPQA-Diamond | 91.9 | 91.7 |
| AA-LCR | 73.0 | 76.3 |
| BFCL (45-item live subset) | 82.2 | — |
| NIAH retrieval @ ~930k-token prompts | 3/3 needles (depths 0.1/0.5/0.9) | — |
| Teacher-forced dNLL vs BF16 (held-out coding traces) | +0.28 nats | — |
| Speculative accept length (EAGLE steps=3, topk=1, draft=4) | ~2.9 | — |
References: Artificial Analysis GLM-5.3. Per-item artifacts are preserved in the repository history.
Serving
python -m sglang.launch_server \
--model-path PhalaCloud/GLM-5.3-W4AFP8 \
--tp 8 \
--quantization w4afp8 \
--disable-shared-experts-fusion \
--speculative-algorithm EAGLE \
--speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
--reasoning-parser glm45 --tool-call-parser glm47
H200-class GPUs serve the full 1M context; on H100 80GB use --mem-fraction-static 0.78
and a reduced context length. vLLM support for this exact checkpoint format is pending
upstream in vllm PR #53848 (--quantization w4afp8 via the humming backend).
Verification
- Structural audit: tensor inventory (176,997 tensors) matches the format-proven GLM-5.2-W4AFP8 checkpoint exactly (names, dtypes, shapes).
- Dequantization audit: int4 expert weights reconstruct within the expected group-128 envelope (10.8-13% relative RMS residual after AWQ channel-scale fitting; no outliers).
- Held-out teacher-forced NLL and task benchmarks: see Evaluation above.
The per-layer AWQ search checkpoints (scales and clip thresholds for all 78 layers) are
preserved in the repository history for reproducibility and further research;
they are kept out of the current revision so snapshot_download fetches only the
serving weights.