DeepSeek-V4-Flash-0731 GGUF

GGUF conversions and expert-focused quantizations of deepseek-ai/DeepSeek-V4-Flash-0731 for llama.cpp.

This repository contains only the practical bullerwins Pareto points retained after size/KLD testing. Models that were dominated or offered only negligible size advantages over a better Unsloth quant were removed.

The routed MoE experts in the lossless reference remain in their native MXFP4 representation. FP8-origin non-expert matrices are dequantized into BF16, which represents their E4M3 values and power-of-two E8M0 scales exactly. Original BF16 and F32 tensors are also preserved. The resulting MXFP4_MOE-BF16 file is lossless relative to the model tensors used by llama.cpp.

Files

The IQ quantizations were generated with the included calibration importance matrix. The mixed autoresearch model keeps all 43 expert down projections in native MXFP4, uses IQ2_XS for the 36 gate/up tensors in layers 0-17, and uses IQ3_XXS for the 50 gate/up tensors in layers 18-42.

KLD quality evaluation

All models were tested against logits from the lossless MXFP4_MOE-BF16 reference using 50 x 512-token Wikitext-2 test chunks. The test evaluated 12,750 output distributions. Lower KLD and higher same-top-token agreement are better. Autoresearch policies were selected on the separate Wikitext-2 validation split; the test split was used only as a holdout check.

Comparison with Unsloth Dynamic quants

Both collections provide strong and closely competitive quality/size tradeoffs. The lossless bullerwins and Unsloth models have byte-identical tensor payloads. The bullerwins IQ2_XS model fills the gap between Unsloth's IQ1 and IQ2 tiers, while bullerwins IQ3_XXS and the 112.07 GiB mixed model provide useful intermediate points between Unsloth UD-IQ3_XXS and UD-Q3_K_M. Unsloth remains stronger around 90, 119, 127, and 144 GiB and also offers smaller sub-80 GiB options. The plot intentionally shows only the bullerwins models retained as practical Pareto points.

Source model

DeepSeek's recommended sampling settings are temperature = 1.0, with top_p = 0.95 for agentic scenarios and top_p = 1.0 otherwise. See the official model repository for architecture details, chat formatting, license, and intended usage.

Original model card

The following model card is from deepseek-ai/DeepSeek-V4-Flash-0731.

DeepSeek-V4-Flash-0731

Introduction

DeepSeek-V4-Flash-0731 is the official release of DeepSeek-V4-Flash, superseding the preview version, with substantially enhanced agentic capabilities. It has the same model structure as DeepSeek-V4-Flash-DSpark, i.e. it comes with a speculative decoding module attached.

DeepSeek-V4-Flash-0731 outperforms DeepSeek-V4-Pro (Preview) on benchmarks listed below despite its far smaller activated parameter count, and is broadly competitive with the strongest proprietary models available.

Notes:

1. For the Code Agent tasks among the public benchmarks above, DeepSeek-V4-Flash-0731 is evaluated with the minimal mode of DeepSeek Harness (to be released) as the agent framework, using the max reasoning effort level with temperature = 1.0, top_p = 0.95.
2. † DSBench-FullStack is an internal full-stack development test set; DSBench-Hard is an internal test set of difficult coding-agent problems.

Chat Template

This release does not include a Jinja-format chat template. Instead, we provide a dedicated encoding folder with Python scripts and test cases demonstrating how to encode messages in OpenAI-compatible format into input strings for the model, and how to parse the model's text output. Please refer to the encoding folder for full documentation.

The reasoning_effort parameter now supports three levels — low, high, and max — which control how much deliberation the model spends before answering.

A brief example:

from encoding_dsv4 import encode_messages, parse_message_from_completion_text

messages = [
    {"role": "user", "content": "hello"},
    {"role": "assistant", "content": "Hello! I am DeepSeek.", "reasoning_content": "thinking..."},
    {"role": "user", "content": "1+1=?"}
]

# messages -> string
prompt = encode_messages(messages, thinking_mode="thinking", reasoning_effort="max")

# string -> tokens
import transformers
tokenizer = transformers.AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-V4-Flash-0731")
tokens = tokenizer.encode(prompt)

How to Run with vLLM

DSpark speculative decoding is enabled with a single flag — add --speculative-config with method: dspark to your vLLM launch command:

--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'

For example, the command below serves the model with vLLM on a single 4×GB300 node. See the vLLM recipe for detailed instructions and other hardware configurations.

vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \
  --trust-remote-code --kv-cache-dtype fp8 --block-size 256 \
  --data-parallel-size 4 --enable-expert-parallel \
  --moe-backend deep_gemm_mega_moe \
  --attention-config '{"use_fp4_indexer_cache": true}' \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'

How to Run with SGLang

Enable DSpark with --speculative-algorithm DSPARK and do not set a separate --speculative-draft-model-path as the target and draft weights therefore come from the same checkpoint. See the SGLang cookbook for detailed instructions, benchmarks and other hardwares configurations.

sglang serve \
  --trust-remote-code \
  --model-path deepseek-ai/DeepSeek-V4-Flash-0731 \
  --tp 4 \
  --moe-runner-backend flashinfer_mxfp4 \
  --speculative-algorithm DSPARK \
  --mem-fraction-static 0.90 \
  --chunked-prefill-size 4096 \
  --swa-full-tokens-ratio 0.1 \

How to Run Locally

Please refer to the inference folder for detailed instructions on running DeepSeek-V4 locally, including model weight conversion and interactive chat demos.

For local deployment, we recommend setting the sampling parameters to temperature = 1.0, with top_p = 0.95 for agentic scenarios and top_p = 1.0 otherwise. For the high and max reasoning effort levels, we recommend a maximum output length of 384K tokens.

License

This repository and the model weights are licensed under the MIT License.

Citation

@misc{deepseekai2026deepseekv4,
      title={DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence},
      author={DeepSeek-AI},
      year={2026},
}

Contact

If you have any questions, please raise an issue or contact us at service@deepseek.com.