DeepSeek V4 Flash Vision Exp GGUF
WARNING: This is not currently supported by llama.cpp master and requires my experimental branch to use https://github.com/tarruda/llama.cpp/tree/dsv4-vision-exp-support
GGUF quantizations for deepseek-ai/DeepSeek-V4-Flash-Vision-Exp.
DeepSeek published the original model weights in MXFP4, so the MXFP4 GGUFs in
this repo are direct conversions of those original safetensors.
Quant Recipes
| Recipe | Quant size | Default type | Expert tensor type |
|---|---|---|---|
Q2_K | 95464.96 MiB (2.82 BPW) | Q8_0 | Q2_K |
IQ3_XXS | 109912.96 MiB (3.24 BPW) | Q8_0 | IQ3_XXS |
Q3_K | 122296.96 MiB (3.61 BPW) | Q8_0 | Q3_K |
For every quant recipe, ffn_down_exps, ffn_gate_exps, and ffn_up_exps use the recipe's expert type. Compressor, indexer, token embedding, output, hyper-connection, router-bias, and token-to-expert mapping tensors retain the BF16, F32, or I32 types specified in scripts/quantize.sh. All three quants use the included 726-entry, 812-chunk importance matrix.
Usage
model="./IQ3_XXS/DeepSeek-V4-Flash-Vision-Exp-IQ3_XXS-00001-of-00004.gguf"
mmproj="./mmproj-DeepSeek-V4-Flash-Vision-Exp-BF16.gguf"
ctx=262144
parallel=1
llama-server --no-mmap --no-warmup \
--model "$model" --mmproj "$mmproj" \
---ctx-size $ctx_size -np $parallel \
--repeat-penalty 1.0 --presence-penalty 0.0 \
--chat-template-kwargs '{"reasoning_effort": "max"}' \
--reasoning-preserve \
-b 2048 -ub 2048 \
--cache-ram 1024 -ctxcp 12 \
--temp 1.0 --top-p 1.0