DeepSeek-V4-Flash-0731 GGUF

GGUF quantizations of deepseek-ai/DeepSeek-V4-Flash-0731, made with llama.cpp.

Chinese version: README_zh.md

Why there is no Q8_0 / Q6_K / Q5_K_M / Q4_K_M

The expert layers ship as FP8 (quantization_config.fmt = e4m3, block 128x128). After conversion to GGUF they land in MXFP4, taking 137.06 GiB (94.1%) of the 156.4 GB master; everything else adds up to roughly 8.6 GiB.

Re-quantizing 4.25 bpw expert weights up to 8.5 bpw only inflates the file — it cannot recover precision that was already discarded upstream. Measured with --dry-run against the same master:

So this repo starts at Q3_K_M. If you want near-lossless, use the original weights directly — an intermediate tier buys you nothing here.

Why there is no IQ2_M

It was built, and it failed testing. Quantized with the same settings, IQ2_M came out at PPL = 1.08e9, while the smaller IQ2_XXS sits at 7.47. Every chunk from the very first one is a 9-digit number, reproducible on both CPU and GPU. Tensor count and the per-layer type map match the working tiers line for line, so this is not a misconfiguration — the IQ2_S code path fails on this model (likely related to imatrix columns left empty by 256-expert top-6 sparse routing). IQ2_XS covers the same size range instead.

Quantizations

The master row is not a file in this repo. It is listed so the numbers above have a reference point: its expert layers are the factory MXFP4 weights with no further quantization, which is the accuracy ceiling for this model.

Every tier is imatrix-calibrated (220 chunks, mixed code / English / Chinese), and the non-expert layers are protected:

Usage

# Plain decoding — fine across multiple GPUs
llama-cli -hf qtum/DeepSeek-V4-Flash-0731-GGUF:Q3_K_M -ngl 99 -c 8192

# Or point at the files yourself
llama-cli -m DeepSeek-V4-Flash-0731-Q3_K_M-00001-of-00004.gguf -ngl 99 -c 8192

For sharded tiers, download all shards and point at -00001-of-*; llama.cpp finds the rest on its own.

DSpark speculative decoding

The base model ships a DSpark speculative-decoding module. This repo exports it separately as dspark-DeepSeek-V4-Flash-0731-BF16.gguf (10.1 GiB, 81 tensors, architecture dflash). One draft file serves every tier.

# --spec-type is required, and everything has to fit on ONE GPU (see below)
CUDA_VISIBLE_DEVICES=0 llama-cli \
  -m DeepSeek-V4-Flash-0731-IQ1_S-00001-of-00004.gguf \
  -md dspark-DeepSeek-V4-Flash-0731-BF16.gguf \
  --spec-type draft-dspark \
  -ngl 99 -ngld 99 -c 2048

Two things worth knowing, both measured here on 8xH100 with llama.cpp 4ed2b13:

• -md on its own does not turn it on. Without --spec-type draft-dspark the draft file loads, the log prints no implementations specified for speculative decoding, and the slot reports "speculative":false — you get plain decoding at full speed and no error. --spec-default does not help either; it only enables ngram-mod and never touches the draft model. With the flag set, the log shows "speculative":true and repeated called impl draft-dspark lines.
• One GPU at a time, for now. Split over 2 or more GPUs it aborts on GGML_ASSERT(n_graph_inputs < GGML_SCHED_MAX_SPLIT_INPUTS) (the cap is 30): the graph wants more cross-backend copies than the scheduler allows. Pinning the draft to one device with -devd does not avoid it, because the target model is what gets split. This is a llama.cpp scheduler limit, not a problem with these files. So the tier plus the 10.1 GiB draft has to fit in a single card — on an 80 GB card that means IQ1_S; the larger tiers need plain decoding until the limit is lifted.

About the PPL numbers

Measured in-house with one fixed setup: wikitext-2 test, n_ctx=512, 12 chunks, -ngl 99, and the master baseline run through the exact same command. These numbers are only comparable within this table. Do not compare them against PPL figures published by other repos — different corpora and chunk counts make the absolute values meaningless across setups.

License

MIT, inherited from deepseek-ai/DeepSeek-V4-Flash-0731. Quantized by qtum.