DeepSeek V4 Flash 0731 - Ombro v3c GGUF

Ombro v3c is an experimental mixed-precision GGUF quantization of the 304B-parameter deepseek-ai/DeepSeek-V4-Flash-0731. It targets consumer systems that must keep much of the routed-expert weights in host memory while preserving the attention path and selected expert layers at higher precision.

Ombro is a community quantization recipe, not an official GGUF quant type. This repository is not affiliated with DeepSeek, Unsloth, or llama.cpp.

File

What changed from Ombro v1

Ombro v1 used very low precision for most routed experts and a higher-precision band at the first and last five transformer layers. V3 first raised all routed experts to IQ2_XXS. V3c retains that uniform baseline while restoring layers 0-4 and 38-42 to IQ3_XXS.

This choice is empirical rather than cosmetic. On the same held-out perplexity set, v3 improved over v1 and restoring the edge layers improved the score again.

Ombro v3 was an intermediate experiment and is not being released because its generation results were not good enough. V3c is the release candidate derived from that line of experiments.

The test used nine 4096-token chunks from a holdout that was excluded from the importance-matrix corpus. These numbers establish a local relative ordering; they are not a claim of parity with the original model or a broad public benchmark result.

Quantization recipe

The official checkpoint was converted to a mixed-precision GGUF intermediate. The final quantization used a newly generated importance matrix covering 49 chunks of 4096 tokens (200,704 evaluated tokens). The calibration material mixed English and German conversation, agentic/code material, and visual coding prompts. The private calibration text is not distributed.

Importance-matrix coverage improved substantially over the matrix used for v1:

DeepSeek V4 Flash has 43 transformer layers. Its attention path, embeddings, router, and shared experts form the always-active backbone. The much larger routed-expert section contains 256 experts per layer, of which only a small subset is selected for each token. Within each expert, gate and up expand and transform the hidden state, while down projects it back to the model width.

The sizes above are derived from the final tensor allocations and rounded to two decimals. They are meant to show where the file size goes; the quant names describe storage formats, not separate downloadable files.

The exact llama-quantize tensor matching rules are:

^token_embd\.weight$=Q8_0
^output\.weight$=Q8_0
^blk\.[0-9]+\.(attn|indexer).*\.weight$=Q6_K
^blk\.[0-9]+\.ffn_(gate|up|down)_shexp\.weight$=Q6_K
^.*hc_.*\.weight$=Q8_0
^blk\.(0|1|2|3|4|38|39|40|41|42)\.ffn_(gate|up|down)_exps\.weight$=IQ3_XXS
^blk\.[0-9]+\.ffn_(gate|up|down)_exps\.weight$=IQ2_XXS

The quantization source, importance matrix, fallback type, and explicit tensor map are preserved. The original one-off v3c shell invocation was not retained, so the public recipe is a reconstruction from those artifacts rather than a claim of byte-for-byte reproducibility across llama.cpp versions.

MTP and speculative decoding

This GGUF contains the 43 main model layers only. The source checkpoint's MTP, NextN, and DSpark draft tensors are not included by the conversion path used for this file. Built-in speculative decoding is therefore unavailable; an external compatible draft model is required if speculative decoding is desired.

Running with llama.cpp

Use a recent CUDA-enabled llama.cpp build with DeepSeek V4 Flash 0731 support. The following example matches a two-GPU consumer system where much of the model must remain memory-mapped from fast local storage:

llama-server \
  --model DeepSeek-V4-Flash-0731-Ombro-v3c.gguf \
  --jinja \
  --ctx-size 131072 \
  --parallel 1 \
  --flash-attn on \
  --cache-type-k bf16 \
  --cache-type-v bf16 \
  --load-mode mmap \
  --no-repack \
  --fit on \
  --split-mode layer \
  --threads 16 \
  --batch-size 2048 \
  --ubatch-size 512 \
  --ctx-checkpoints 0 \
  --cache-ram 0 \
  --temp 1.0 \
  --top-p 0.95

Adjust GPU offload, thread count, context, and batch sizes for your hardware. Keep K and V cache types identical. bf16 is the recommended KV-cache type for Ombro v3c: it produced the most dependable local results and is the default used for validation. q8_0 also produced a usable manually reviewed result, but the available Q8 testing is too limited and not prompt-matched closely enough to claim equivalent quality. Use it when reducing KV-cache memory is more important than staying on the best-tested path.

DeepSeek recommends temperature=1.0, with top_p=0.95 for agentic scenarios and top_p=1.0 for other workloads. Do not add sampler values merely because a frontend exposes them; explicit zero is not the same as an unset default.

Validation status

• Loads and generates successfully with llama.cpp.
• Perplexity measured on a held-out local corpus as shown above.
• Local runtime tuning and generation tested at up to 128k context.
• No formal long-context accuracy, KL-divergence, knowledge, coding, agentic, or source-model equivalence benchmark has been completed.
• Visual one-shot benchmarks are useful smoke tests but are not treated as sufficient evidence of general model quality.

Local visual smoke-test summary

Across three shared visual prompt families, the mean local Auto-Eval score was 53.3/100 for Ombro v1 and 67.3/100 for Ombro v3c. These checks primarily measure whether generated applications load, render, move, and avoid runtime errors; they do not directly score reasoning quality, code quality, or visual fidelity. Run-to-run variance was substantial, so this aggregate is included only as a smoke-test signal. The held-out perplexity result above is the cleaner controlled comparison between the quantization recipes.

License and attribution

The original model and weights are released under the MIT License. See the DeepSeek model repository for the original model card, technical information, intended use, limitations, and license terms.