GLM-5.3-NVFP4

NVFP4 quantization of Z.ai's zai-org/GLM-5.3, produced from the BF16 release (zai-org/GLM-5.3-BF16) with NVIDIA Model Optimizer. Run it on Modal.

• Base model: zai-org/GLM-5.3-BF16 (revision 861dfa23)
• Quantization: NVFP4 weights and activations on the routed expert linears, FP8 KV cache
• Producer: Model Optimizer 0.47.0.dev (upstream 5500999d0), general/ptq/nvfp4_experts_only-kv_fp8_cast recipe
• Size: 465 GB (47 shards)

Quantization Method

Only the linear operators of the routed MoE experts (gate_proj, up_proj, down_proj in model.layers.{3..77}.mlp.experts.*) are quantized, to NVFP4 (4-bit float, group size 16, two-level scaling: per-16-element FP8 block scales plus a per-tensor global scale). Expert activations use NVFP4 with static, max-calibrated per-expert input scales.

Kept in BF16: all attention (MLA projections and the DSA indexer), shared experts, the first three dense-MLP layers (model.layers.{0..2}), the MTP/NextN layer (model.layers.78, usable for speculative decoding), lm_head, and embeddings. The KV cache is FP8 (E4M3) in cast mode (unit scale, no data-driven KV calibration). The full per-layer quantizer map is in .quant_summary.txt. The exclusion list is in hf_quant_config.json.

Calibration

Calibrated with max calibration on 1,024 samples at sequence length 1,024. The calibration set covers general text, chat, code, software engineering, agentic tool use, and math: cnn_dailymail (256 samples), Nemotron-SFT-Instruction-Following-Chat-v2 (192), OpenCodeReasoning (256), Nemotron-SFT-SWE-v2 (128), Nemotron-SFT-Agentic-v2 (128), and Nemotron-Math-v2 (64). Chat data was formatted with the model's chat template, with tool definitions included.

Deployment

Run it on Modal. Use the model's recommended sampling (temperature=1.0, top_p=0.95).

License

Quantized version of GLM-5.3, distributed under Z.ai's GLM-5.3 License, inherited from the base model.