DeepSeek-V4-Flash-0731-GPTQ-Int4-V1
4-bit GPTQ (W4A16) quantization of deepseek-ai/DeepSeek-V4-Flash-0731, produced with GPTQModel.
This is the V1 release (first calibration run). A V2 release, requantized with a
different calibration dataset, is planned as a separate repository
(DeepSeek-V4-Flash-0731-GPTQ-Int4-V2) once available.
Quantization details
| Method | GPTQ (GPTQModel 7.3.4) |
| Bits | 4 |
| Group size | 128 |
| Symmetric | yes |
desc_act | false |
lm_head quantized | no |
| Pack dtype | int32 |
Calibration data
128 samples (32 each of zh / en / ja / code), seq_len=2048:
allenai/c4— zh, en, ja splits (mangled/mis-decoded documents filtered out)codeparrot/codeparrot-clean-valid— Python source
zh/en/ja/code were weighted evenly. seq_len=2048 was chosen above this model's
sparse-indexer index_topk (512) so the top-k pruning path is actually exercised
during calibration, while staying within the indexer's dense-attention memory budget
on the quantization rig.
Serving with vLLM
This model requires a vLLM build with DeepSeek-V4 GPTQ support on the Ampere
(SM86, e.g. RTX 3090/4090-class, non-Hopper) backend, which is not yet present in
either upstream vLLM or the unmodified vllm-dsv4-ampere community fork. The
required patch is available at:
https://github.com/okdshin/vllm-dsv4-ampere/tree/gptq-w4a16-support
Apply the main-port/vllm overlay from that branch on top of a vanilla vLLM
checkout, then serve as usual, e.g.:
vllm serve sokada4/DeepSeek-V4-Flash-0731-GPTQ-Int4-V1 \
--trust-remote-code \
--kv-cache-dtype fp8 \
--tensor-parallel-size 4 \
--enable-expert-parallel \
--enforce-eager \
--max-num-seqs 8 \
--max-model-len 4096
(--cpu-offload-gb may be needed depending on available VRAM; the full checkpoint
is ~153GB.)
Benchmark
Measured on a 4x GPU Ampere rig (--cpu-offload-gb 20, vllm bench serve),
compared against the unquantized fp8 checkpoint:
| concurrency | metric | fp8 (unquantized) | GPTQ 4bit | delta |
|---|---|---|---|---|
| 1 | TTFT mean | 2345.56 ms | 1184.35 ms | -49% |
| 1 | TPOT mean | 335.33 ms | 317.98 ms | -5% |
| 1 | output tok/s | 2.73 | 3.02 | +11% |
| 8 | TTFT mean | 8434.68 ms | 4486.18 ms | -47% |
| 8 | TPOT mean | 735.13 ms | 432.72 ms | -41% |
| 8 | output tok/s | 9.34 | 16.11 | +72% |
At concurrency=1, decode is bound by PCIe transfer of CPU-offloaded weights rather than GPU compute/bandwidth, so the TPOT improvement from quantization is modest; at concurrency=8, batching amortizes that fixed cost and the quantization benefit is much larger.
License
MIT, inherited from the base model.