GLM-5.3-Flash-NVFP4
Model Overview
- Model Architecture: Glm5NextForConditionalGeneration
- Input: Text / Image
- Output: Text
- Model Optimizations:
- Weight quantization: FP4
- Activation quantization: FP4
- Release Date: 2026-08-27
- Version: 1.0
- Model Developers: RedHatAI
This model is a quantized version of zai-org/GLM-5.3-Flash. It was evaluated on several tasks to assess its quality.
Model Optimizations
This model was obtained by quantizing the MoE expert weights and activations of zai-org/GLM-5.3-Flash to FP4 (NVFP4) data type, ready for inference with vLLM. The MTP layers are kept in FP8, matching the source checkpoint.
This optimization reduces the number of bits per parameter in the quantized layers from 8 to 4, reducing the disk size and GPU memory requirements by approximately 50% of the quantized weights.
Only the weights and activations of the MoE expert linear operators are quantized using LLM Compressor.
Deployment
vLLM Serving
docker run --gpus all \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
vllm/vllm-openai:glm53-flash RedHatAI/GLM-5.3-Flash-NVFP4 \
--tensor-parallel-size 4 \
--no-enable-flashinfer-autotune \
--tool-call-parser glm47 \
--enable-auto-tool-choice \
--reasoning-parser glm45
Enable Speculative Decoding
docker run --gpus all \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
vllm/vllm-openai:glm53-flash RedHatAI/GLM-5.3-Flash-NVFP4 \
--tensor-parallel-size 4 \
--no-enable-flashinfer-autotune \
--tool-call-parser glm47 \
--enable-auto-tool-choice \
--reasoning-parser glm45 \
--gpu-memory-utilization 0.85 \
--disable-custom-all-reduce \
--speculative-config '{"method":"mtp","num_speculative_tokens":5}'
Creation
This model was created by applying LLM Compressor with the NVFP4 scheme, exported in compressed-tensors format.
Evaluation
This model was evaluated on GSM8K Platinum, MATH-500, AIME 2025, and GPQA Diamond using lm-evaluation-harness and lighteval, all served with vLLM (OpenAI-compatible API). Each benchmark was run with 3 seeds (8 seeds for AIME 2025) and the results averaged.