DeepSeek-V4-Flash-MXFP4
Model Overview
- Model Architecture: DeepseekV4ForCausalLM
- Input: Text
- Output: Text
- Supported Hardware Microarchitecture: AMD MI355 / MI350 (gfx950)
- ROCm: 7.2.0
- PyTorch: 2.9.1
- Transformers: 5.13.1
- Operating System(s): Linux
- Inference Engine: vLLM
- Model Optimizer: AMD-Quark (v0.12.0)
- Quantized layers: All routed + shared MoE expert projections. All other modules (attention, the MoE router gate, norms, embeddings, the output head, and the MTP block) are excluded and kept in original precision.
- Weight quantization: OCP MXFP4, Static
- Activation quantization: OCP MXFP4, Dynamic
Model Quantization
Quantized from deepseek-ai/DeepSeek-V4-Flash with AMD Quark. The pipeline re-quantizes only the MoE expert weights and activations to MXFP4. All non-expert modules are kept as-is via the exclude list.
Quantization script
from quark.torch import ModelQuantizer
from quark.torch.quantization.config.template import LLMTemplate
template = LLMTemplate.get('deepseek_v4')
qconfig = template.get_config(scheme='mxfp4')
ModelQuantizer(qconfig).direct_quantize_checkpoint(
pretrained_model_path='<DSV4_Flash_src_path>',
save_path='<output_dir>',
keep_excluded_layers_as_original_model_state=True,
)
Deployment
Use with vLLM
This model can be deployed efficiently using the vLLM backend based on the Docker image vllm/vllm-openai-rocm:v0.29.0. vLLM and lm_eval are both installed from source.
Evaluation
The model was evaluated on gsm8k (8-shot) benchmark using the vLLM framework.
Accuracy
| Benchmark | deepseek-ai/DeepSeek-V4-Flash | amd/DeepSeek-V4-Flash-MXFP4(this model) | Recovery |
|---|---|---|---|
| GSM8K (flexible-extract) | 95.00 | 94.92 | 99.9% |
Reproduction
The GSM8K results were obtained using the lm-eval framework, based on the Docker image vllm/vllm-openai-rocm:v0.29.0.
Launching server
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1
vllm serve amd/DeepSeek-V4-Flash-MXFP4 --tensor-parallel-size 4 --kv-cache-dtype fp8 \
--trust-remote-code --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4 \
--tool-call-parser deepseek_v4 --enable-auto-tool-choice \
--compilation-config '{"mode": 3, "cudagraph_mode": "FULL_DECODE_ONLY"}'
Evaluating model in a new terminal
lm_eval --model local-completions \
--model_args model=amd/DeepSeek-V4-Flash-MXFP4,base_url=http://localhost:30000/v1/completions,tokenized_requests=False,num_concurrent=32 \
--tasks gsm8k --batch_size auto --num_fewshot 8
License
This model is a quantized derivative of deepseek-ai/DeepSeek-V4-Flash and is distributed under the same license as the source model: the MIT License. A copy of the upstream LICENSE is included in this repository.
Modifications Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved. AMD has modified the model weights of the MoE expert layers by quantizing them to MXFP4 with AMD Quark; the modifications are provided under the same MIT License and are not subject to any separate or different license.