DeepSeek-V4-Flash-0731 — NVFP4 routed experts (incl. MTP)
deepseek-ai/DeepSeek-V4-Flash-0731
(revision 7872f01b) with all 256 routed experts per MoE layer quantized to
NVFP4 W4A4 — including, since the 2026-08-03 update, the routed experts of
the three MTP (multi-token-prediction) layers — produced with NVIDIA
Model-Optimizer's
DeepSeek-V4 routed-expert recipe. Everything else — attention, router gates,
shared experts, embeddings, and lm_head — is untouched from the source
release (FP8 per quant_method: fp8).
Built for Blackwell GPUs (sm_120+), where NVFP4 GEMMs run on native FP4 tensor cores. Serves with vLLM / TensorRT-LLM builds that support the DeepSeek-V4 hybrid FP8 + NVFP4-MoE layout; verified under vLLM at TP=2 on 2× RTX PRO 6000 Blackwell (96 GB).
2026-08-03 update: NVFP4 MTP experts → working speculative decoding
The initial upload left the MTP layers' expert weights in the source MXFP4
format, which does not load correctly through the NVFP4 fused-MoE path — MTP
speculative-decoding acceptance collapsed to ~15% and spec decode was a net
loss. This update recasts the routed-expert weights of all 3 MTP layers to
NVFP4 with the same closed-form lossless cast: 2,304 tensors,
603,979,776 / 603,979,776 blocks lossless (100.0%), max dequant error 0.0
(see mtp_nvfp4_build_report.json). Changed files: shards 46–48, the index,
config.json, and hf_quant_config.json (recipe version
dsv4-nvfp4-experts-mtp-fallback).
With this checkpoint MTP speculative decoding (DSpark) works, roughly doubling single-stream decode throughput.
Benchmarks
vLLM, TP=2 on 2× RTX PRO 6000 Blackwell (96 GB), PCIe (no P2P), 300 W/GPU
power cap, 131k context. 512-token greedy completions, temp 0, unique prompt
nonces; acceptance measured from vLLM /metrics counter deltas.
| Metric | MTP off | MTP spec decode (DSpark, 5 draft tokens) |
|---|---|---|
| Single-stream decode, median | 112.3 tok/s | 199.9 tok/s (1.78×) |
| Single-stream decode, mean | 112.3 tok/s | 220.2 tok/s (1.96×) |
| Single-stream range | 112.2–112.9 | 167–302 (content-dependent) |
| 2-stream aggregate | — | 353 tok/s |
| 4-stream aggregate | — | 411 tok/s |
| TTFT, ~1.8k-token prompt, streaming | — | 253–258 ms |
A second suite on the identical config measured median 242.5 / mean 232.7 tok/s — greedy tie-breaks change the generated text between sessions, so single-stream medians move ±20% run to run. Typical: ~200–240 tok/s.
Acceptance: mean acceptance length 3.85 tokens/step (of 6 max), cumulative draft acceptance 49.4% — in line with DeepSeek's published DSpark figure (50.8%). Throughput tracks acceptance roughly linearly (accept 2.8 → ~168 tok/s; accept 5.6 → ~302 tok/s); the spread is content predictability, not jitter.
Recommended spec-decode settings
- Greedy draft sampling. Probabilistic draft sampling garbles output through this NVFP4 path; greedy is clean at all target temperatures (verified at T = 0 / 0.7 / 1.0, including thinking mode).
- 5 draft tokens, fixed depth. Depth 7 is slower (positions 6–7 accept only 1–10% and the verify cost is paid every step); the dynamic-depth controller oscillates at single-stream (166–177 tok/s).
- bf16 draft head. An FP8 draft head costs acceptance (3.51 vs 4.09 accept length; 196 tok/s median).
Why the weights are bit-exact
The source release already ships routed experts as MXFP4 (E2M1 nibbles +
E8M0 power-of-two scale per 32-element block). Instead of the usual lossy
dequantize→requantize round trip, this conversion uses a closed-form cast:
weight_scale_2 is pinned to 2^(k_max − 8) and each 16-element block's E4M3
scale to 2^(k_j − m), so every NVFP4 nibble equals the source MXFP4 nibble
verbatim (w1/w3 share one scale_2 for the fused GEMM1).
Result: 8,657,043,456 / 8,657,043,456 main-layer blocks and 603,979,776 / 603,979,776 MTP-layer blocks lossless (100.0000%) — the expert weights are byte-faithful to the original release. Quantization only introduces the calibrated activation scales.
Calibration
input_scale (per-tensor activation scale for W4A4) comes from post-training
calibration with Model-Optimizer:
- Data: 64 samples each from
abisee/cnn_dailymailandnvidia/Nemotron-Post-Training-Dataset-v2(stem/chat/math/code splits) - Setup: MP=2 over 2× RTX PRO 6000 Blackwell, batch size 4
- Experts that received no tokens during calibration fall back to the max observed scale among calibrated experts of the same projection; MTP-layer experts use the same fallback scheme
Layout
Original HF 48-shard layout. Per routed-expert weight (main and MTP layers),
the MXFP4 .scale sibling is dropped and three keys are added:
| key | contents |
|---|---|
…weight | NVFP4-packed uint8, shape (out, in/2) |
…weight_scale | per-16-block E4M3 scale, shape (out, in/16) |
…weight_scale_2 | per-tensor FP32 scale |
…input_scale | per-tensor FP32 activation scale |
config.json keeps the source FP8 quantization_config and adds
moe_quant_algo: "NVFP4" (group size 16); hf_quant_config.json carries the
full quantized-layer manifest (43 MoE layers + mtp.{0,1,2}.ffn.experts).
MTP shared-expert weights remain in the source format.
Provenance
- Base:
deepseek-ai/DeepSeek-V4-Flash-0731@7872f01b(304B params, 43 layers, 256 routed experts, 3 MTP layers) - Tooling: nvidia-modelopt 0.45.0, torch 2.13.0+cu130, safetensors 0.8.0
- License: MIT, inherited from the base model (see
LICENSE)