DeepSeek-V4-Flash-Vision-Exp REAP-145B — one 96 GB GPU

DeepSeek-V4-Flash-Vision-Exp is 284B parameters in 167.8 GB — the experts ship 4-bit (MXFP4) from the source, so further quantization has nothing left to take, and no single-GPU artifact existed. This one fits: half the routed experts pruned (K128 of 256, REAP), 83.3 GB, serves on one 96 GB card with stock vLLM 0.28.0. Expert bytes are untouched source MXFP4 — pruning removes experts, it never requantizes the survivors.

The expert selection transplants ludo-tech's REAP plan (computed on the text-only 0731 base with their published toolchain, MIT) onto the Vision-Exp weights; the frozen hash-router tables (tid2eid, layers 0–2) are remapped from this checkpoint's own gate tensors. Two Vision-Exp-only gate tensors the text runtime cannot load (gate.bias on hash layers, gate.bias_vl everywhere) are sliced to the kept experts and stored in vision_tower.safetensors with the vision tower and projector — shipped for the day vLLM grows deepseek_v4 vision support, outside the weight index until then. MTP is dropped (inherited from the REAP plan). Text-only serving today, by runtime necessity: vLLM has no vision path for this architecture yet.

Serve (vLLM 0.28.0, one 96 GB Blackwell card)

Recommended (CUDA graphs, 20K context — 73.1 tok/s single-stream; the model reasons long, so the full context budget matters):

vllm serve primitive-ai/DeepSeek-V4-Flash-Vision-Exp-REAP-145B \
  --max-model-len 20480 --gpu-memory-utilization 0.95 \
  --max-num-batched-tokens 2048 --max-num-seqs 4 \
  --kv-cache-dtype fp8

Batch throughput (eager, concurrency 16 — 150–172 tok/s aggregate):

vllm serve primitive-ai/DeepSeek-V4-Flash-Vision-Exp-REAP-145B \
  --max-model-len 20480 --gpu-memory-utilization 0.95 \
  --max-num-batched-tokens 4096 --max-num-seqs 16 \
  --kv-cache-dtype fp8 --enforce-eager

Every flag is load-bearing on 96 GB:

• --kv-cache-dtype fp8 — the architecture's fp8_ds_mla KV layout accepts nothing else.
• --gpu-memory-utilization 0.95 with tight --max-num-batched-tokens / --max-num-seqs — weights take 83.3 GB; the batching bound shrinks activation workspace enough to fit the context and, in the first config, the CUDA graphs. At vLLM's default batching, even 8K does not fit. Graphs are worth 5× on single-stream decode here (75.7 vs 13.9 tok/s eager): this architecture launches many small kernels per token.
• vLLM 0.28.0 or newer. Older builds (including recent dev snapshots) fail on RTX-class Blackwell (SM120) at five distinct layers: DeepGEMM's MegaMoE and hyper-connection kernels assert (they require SM100), cutlass c3x has no SM120 block-FP8 GEMM, the triton fallback lacks the e8m0 scale dtype, and tilelang 0.1.12 aborts on import next to apache-tvm-ffi (0.1.13 fixes that). 0.28.0 routes all of it correctly out of the box.

Measured

One RTX PRO 6000 Blackwell (96 GB). Our standard 1,170-item knowledge + 200-item tool-calling suites, temperature 0.6 / top_p 0.95 / top_k 20, thinking on, 16,384-token budget, auto-scored. Suites ran at concurrency 16 in the eager 20K configuration (our other cards run 32; the memory-bound --max-num-seqs cap decides here).

Highlights: gsm8k 100.0, arc_challenge 90.7, mmlu_pro 90.8 completed-only. The strict/completed gap is verbosity, not wrongness: mean answer runs 2,761 tokens and 8.8% of items hit the 16,384-token budget — use the 20K serve config if that matters to you.

The KV cache is DeepSeek's fp8_ds_mla compressed-latent format — the architecture accepts no other layout (4-bit KV schemes like TurboQuant don't apply here; there is far less cache to squeeze than on a GQA model to begin with).

No unpruned baseline column: the full model cannot serve on this hardware, which is the point of this artifact — treat the numbers as absolute, not as a delta. Zero request errors across both suites (4.3 h of continuous serving).

What pruning half the experts looks like

Call accuracy held up (81.9 beats several unpruned models on our same-box leaderboard); abstention is the visible casualty (45.0 — the model rarely declines to call a tool). If your workload punishes spurious tool calls, weight that column.

Vision serving — working today, on a stock nightly

Two of the three upstream pieces have landed since this section was first written: vLLM's vision support for the architecture (#54566) merged on 2 September, and FlashInfer's SM120 sparse-MLA refactor (#4802) merged on 3 September. The third, vllm#54631, has shrunk to a three-line DSpark speculative-config fix that this recipe does not exercise. So the recipe is now short. This checkpoint serves images on one 96 GB card, validated end to end on 6 September with the steps below (shape and colour identification, object counting, text unaffected on the same serve). To our knowledge this is still the only single-GPU vision serving of this model; the official test hardware is 4×GB200.

1. Copy vision_serving/model.safetensors.index.json and vision_serving/config.json over the checkpoint's own (they index the vision tensors back in and set DeepseekV4ForConditionalGeneration). Keep the originals to switch back to text-only serving.

2. Use a vLLM main build (vllm/vllm-openai:nightly, any build at or after 1356635d) and install FlashInfer from its nightly index at dev20260904 or later, which ships the merged SM120 kernel prebuilt. No released FlashInfer carries it yet: 0.6.18 and 0.6.18.post1 both predate the merge.

   FROM vllm/vllm-openai:nightly
   RUN pip install --no-cache-dir --pre --upgrade \
         --extra-index-url https://flashinfer.ai/whl/nightly \
         --extra-index-url https://flashinfer.ai/whl/nightly/cu130 \
         "flashinfer-python==0.6.18.dev20260904" \
         "flashinfer-jit-cache==0.6.18.dev20260904+cu130" \
         "flashinfer-cubin==0.6.18.dev20260904"
   

3. Serve with --block-size 256 --attention-backend FLASHINFER_MLA_SPARSE_DSV4 --tokenizer-mode deepseek_v4 --kv-cache-dtype fp8 --no-enable-flashinfer-autotune --limit-mm-per-prompt '{"image":2,"video":0}' plus the memory flags from above (--max-model-len 32768 --gpu-memory-utilization 0.95 --max-num-seqs 8 --enforce-eager). First boot warms up DeepGEMM kernels (302 s to healthy here with an empty cache here); mount /root/.cache to keep them.

No vLLM overlays and no JIT-cache mask on this path. The vision_serving/vllm_overlays/ files stay in the repo for the pre-merge vllm/vllm-openai:deepseekv4-flash-vision branch image only; do not mount them on a main build, where they would revert newer fixes to the same files. Vision quality of a half-pruned model is smoke-tested, not benchmarked; treat it as experimental.

Provenance

• Base: deepseek-ai/DeepSeek-V4-Flash-Vision-Exp (MIT), experts MXFP4 + FP8 attention as shipped.
• REAP plan and rewriter: ludo-tech/DeepSeek-V4-Flash-REAP-145B-A13B tools/v4prune (MIT) — the transplant recomputes only the hash-table remaps against this checkpoint's router.
• Files: 45 weight shards, numbered out of 48 (83.3 GB with the tower) + vision_tower.safetensors (vision tower, aligner, image-token embeddings, sliced gate.bias_vl) + tokenizer/config.