THIS IS A WIP (Work In Progress)

DeepSeek V4 Flash · GGUF

GGUF quantizations of deepseek-ai/DeepSeek-V4-Flash for use with the V4-aware llama.cpp fork at cchuter/llama.cpp @ feat/v4-port-cuda.

📦 Required: V4-aware llama.cpp fork. These quants don't load on upstream ggml-org/llama.cpp — V4 architecture support (compressor decode, hyperconnection, lightning indexer, FP8 KV simulation, NextN heads) lives only in the fork:

git clone -b feat/v4-port-cuda https://github.com/cchuter/llama.cpp

Full build + run instructions in Loading below.

🖥️ Supported backends: Apple Silicon (Metal), NVIDIA CUDA (Ada/Blackwell), and CPU. All 5 V4 custom ops (ggml_dsv4_rope_tail, ggml_dsv4_hc_split_sinkhorn, ggml_dsv4_hc_weighted_sum, ggml_dsv4_hc_expand, ggml_dsv4_fp8_kv_quantize) have Metal kernels AND CUDA kernels in this fork (validated 19/19 on RTX 5090, CUDA 12.8, SM_120 native). The CUDA FP8 path is gated behind __CUDA_ARCH__ >= 890; older NVIDIA hardware (Volta/Turing/Ampere) uses a software-emulated FP8 path that builds cleanly under -DCMAKE_CUDA_ARCHITECTURES=70 but hasn't been runtime-validated yet. CUDA testers wanted — file issues at the fork if you hit problems. ROCm / Vulkan / Metal-on-AMD have no V4 kernels and will fail at the first dsv4 op.

Available quants

† All quants in this repo ship with the DSML chat template baked into the GGUF metadata, so llama-server --jinja does the right thing without any extra flags. The imatrix/dsml.jinja file is also published in this repo for reference and downstream tooling.

-XL suffix means non-expert tensors (output_tensor, token_embd, attention projections, attention compressors, hyper-connection mixers, lightning indexer, NextN heads) are pinned at Q8_0; only the routed and shared experts use the named quant body. Without that pinning, IQ-class quants fall below the 16 t/s decode floor on M3 Ultra.

Recommended use by quant

All quants in this repo ship with V4's DSML chat template baked in, so llama-server --jinja does the right thing without any extra flags — no --chat-template-file needed. Tool calls return as proper tool_calls JSON in the response object.

Loading

# Clone the V4-aware fork
git clone -b feat/v4-port-cuda https://github.com/cchuter/llama.cpp
cd llama.cpp

# Build for Apple Silicon (Metal)
cmake -B build -DGGML_METAL=ON -DGGML_METAL_EMBED_LIBRARY=ON && cmake --build build -j

# OR build for NVIDIA CUDA. Pick your GPU's compute capability:
#   sm_70 V100 | sm_75 T4 | sm_80 A100 | sm_86 RTX 3090/3080
#   sm_89 RTX 4090/6000 Ada/L40 | sm_90 H100/H200 | sm_120 RTX 5090/5080
# (List multiple if you ship to mixed hardware, e.g. "86;89".)
# FP8 native path needs SM_89+ (Ada/Hopper/Blackwell) AND CUDA toolkit >= 11.8;
# older arches use the software-emulated FP8 path automatically. SM_120 native
# additionally needs toolkit >= 12.8 (older toolkits fall back to PTX JIT).
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CUDA_ARCHITECTURES="<your-sm>" && cmake --build build -j

# Multi-GPU CUDA (2+ devices): pass the SCHED flag to BOTH compiler groups so
# the macro propagates to .cu translation units. CXX-only is silently no-op
# on the CUDA side. V4's dense per-layer inputs (hyperconnection + indexer +
# multiple KV caches) exceed the upstream scheduler default of 30 at
# multi-device split boundaries. Cost: ~200 MB extra scheduler memory; only
# needed on multi-GPU. Single-GPU runs do not need this flag.
# cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release \
#   -DCMAKE_CUDA_ARCHITECTURES="<your-sm>" \
#   -DCMAKE_CXX_FLAGS=-DGGML_SCHED_MAX_SPLIT_INPUTS=128 \
#   -DCMAKE_CUDA_FLAGS=-DGGML_SCHED_MAX_SPLIT_INPUTS=128

# Download the recommended Q4_K_M-XL shards
hf download teamblobfish/DeepSeek-V4-Flash-GGUF \
  --include "Q4_K_M-XL/*" \
  --local-dir ~/models/DeepSeek-V4-Flash-GGUF

# Run the server (point at the first shard; llama.cpp auto-loads the rest)
./build/bin/llama-server \
  --model ~/models/DeepSeek-V4-Flash-GGUF/Q4_K_M-XL/DeepSeek-V4-Flash-Q4_K_M-XL-00001-of-00004.gguf \
  --jinja \
  --reasoning off \
  --ctx-size 393216 \
  --n-gpu-layers 999 \
  --flash-attn on \
  --no-repack \
  --temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0

Sampling values match the model card recommendation (temperature=1.0, top_p=1.0); --reasoning off is the cleanest baseline for agent workloads.

Multi-GPU CUDA (work in progress)

⚠️ Status: WIP. Multi-GPU CUDA via --split-mode layer (the default) is working end-to-end and validated on 2× RTX 6000 Ada (sm_89, 96 GB total) at the speeds in the table below, with an external tester also reporting it working on 8× A100. Tensor-parallel (--split-mode row) is implemented but currently slower than layer split for V4 decode and not recommended yet. Expect quirks; please file issues at the fork.

Recommended config for fastest t/s on multi-GPU:

./build/bin/llama-server \
  --model ~/models/DeepSeek-V4-Flash-GGUF/<quant>/<first-shard>.gguf \
  --jinja --reasoning off \
  --ctx-size 8192 \
  --n-gpu-layers 999 \
  --split-mode layer \
  --flash-attn on \
  --no-repack \
  --temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0

Pick a quant that fits your combined VRAM (e.g. IQ2_XS-XL at 81 GiB fully fits 96 GiB across 2× 48 GB). If the quant doesn't fully fit, add -cmoe -ub 128 to offload routed experts to CPU — fits much larger quants at a generation-speed cost.

Validated speeds (IQ2_XS-XL, 2× RTX 6000 Ada):

Why the -XL recipe (and why no vanilla Q4_K_M)

V4 decode is compute-bound on the indexer / sinkhorn / expert-routing kernels — not on memory bandwidth. That makes the choice of dequant codepath matter as much as the bit-count: Q8_0's int8 × per-block-scale unpack is dramatically simpler than Q4_K_M's super-block path, so on this hardware Q8_0 actually decoded faster than vanilla Q4_K_M in our earlier benchmarks (write-up).

The -XL recipe published here threads that needle: leave the discrimination-critical non-expert tensors at Q8_0 (so attention, embedding, output, etc. all use the fast dequant path) and only compress the routed and shared experts. The result is the best of both — Q4_K_M-XL is half the disk of Q8_0 with essentially identical decode speed (22.85 vs 21.69 t/s) because the experts barely touch the hot decode path while the bandwidth-heavy non-expert tensors stay on the fast codepath. Same trick applies to all the IQ-class -XL variants below.

We don't publish vanilla Q4_K_M (no XL pins) — it would be both larger and slower than Q4_K_M-XL on this hardware.

Quirks worth knowing

• --cache-type-k|v q8_0 is silently overridden to f16 on V4. V4's K is already FP8-quantized at write time, so q8_0's per-block stationarity assumption breaks. The fork emits a LLAMA_LOG_WARN on first override.
• llama-imatrix originally segfaulted on V4 during activation collection. Fixed in v4-port-I-imatrix; the calibration data published alongside these quants (imatrix/imatrix-v4-flash.dat) was produced by the patched binary.
• --no-repack is required for V4 quants in CPU mode on hosts smaller than ~600 GiB RAM. The repack codepath in ggml/src/ggml-cpu/repack.cpp doesn't release the source mmap, so V4's 282-GiB Q8 source needs ~575 GiB peak RAM at load without the flag. The fork's gates pass --no-repack by default.
• Validation gates: tests/v4-port/run-all-gates.sh in the fork. Each row in the table above documents the result of that gate suite at the listed BPW.

Provenance

• Source: deepseek-ai/DeepSeek-V4-Flash HF safetensors (FP8 e4m3 weights, FP4 routed experts).
• Q8_0: built via convert_hf_to_gguf.py --outtype q8_0 --deepseek4-expert-outtypes q8_0 (M3 Ultra, ~30–60 min wall), split into 50 GiB shards with llama-gguf-split.
• bf16-experts-Q8 staging GGUF (not published): built via convert_hf_to_gguf.py --outtype bf16 --deepseek4-expert-outtypes q8_0. Used as the source for IQ1/IQ2/Q2_K-XL/Q4_K_M-XL builds below to preserve embed.weight and output.weight BF16 source precision (other discrimination-critical tensors are FP8-native in the source so Q8 staging is essentially lossless for them).
• IQ1/IQ2/Q2_K-XL/Q4_K_M-XL builds: produced via llama-quantize --imatrix imatrix-v4-flash.dat with the v4-port fork's V4-tensor pin recipe (output_hc, attn_compressor, attn_q_a/b, attn_kv, attn_output_a/b, hc_attn, hc_ffn, indexer, nextn all at Q8_0 in -XL variants).
• imatrix: wikitext-103 test split, 1000 chunks, ~1M tokens. Per-class layer coverage verified by tests/v4-port/gate-imatrix.sh.

License

MIT, matching the upstream DeepSeek V4 Flash license.