DeepSeek-V4-Flash-0731 — AWQ 2/3-bit (MLX / oMLX)

AWQ-calibrated mixed-precision quantization of DeepSeek-V4-Flash-0731 (304B MoE, 43 layers, 256 routed experts/layer, Hyper-Connections, HISA attention) for Apple Silicon via oMLX.

Routed experts carry the aggressive quantization; everything else stays at 8-bit.

Backbone: 278.6B quantized params · 101.4 GB · BPW 2.91 oMLX build with quantized MTP drafter: 108.2 GB total (drafter 7.04 GB, down from 10.86 GB unquantized)

The MTP/DSpark speculative drafter is quantized to the same 2/3-bit recipe, not left at native precision. BPW above is quoted for the backbone weights; the drafter is additional and is not part of the 278.6B count.

Recommended sampling

temperature = 1.0
top_p       = 0.95
min_p       = 0.05

For long-context work, enable KV cache quantization at 4-bit or 6-bit — this model's context cost is dominated by KV, and 6-bit is essentially free quality-wise while 4-bit buys substantially more usable context.

reasoning_effort supports low, high, max (not medium/xhigh — the chat template rejects those). Pass via chat_template_kwargs.

Performance (M5 Max, 128 GB, oMLX, MTP enabled)

Single request

Generation speed is essentially flat with context — TPOT stays at ~20–22 ms from 1k to 16k, i.e. ~47–51 tok/s throughout. The MTP/DSpark drafter is doing its job.

Batched

Two things worth knowing

Build the custom kernels for long context. If oMLX reports dsa_indexer_scores/dsa_topk_indices unavailable (glm_moe_dsa extension not built), the sparse-attention indexer falls back to a slow MLX path and long-context prefill is several times slower than it needs to be. Rebuild with OMLX_WITH_CUSTOM_KERNEL=1. Generation speed is unaffected — this is purely a TTFT issue, and it is the largest single speedup available.

Memory and context. KV growth is modest (~4 GB from 1k to 16k), but the weights alone occupy ~102 GB, so headroom runs out well before very long contexts on a 128 GB machine. Use 4-bit or 6-bit KV cache quantization for long-context work — 6-bit is effectively free, 4-bit buys substantially more usable context.

Measured results

Benchmarks via oMLX's own omlx.eval harness (n=200 MMLU/GSM8K, n=164 HumanEval), MTP enabled:

The v2 recalibration used a broader, multi-turn agentic calibration set plus down_proj at group size 64. Both changes were necessary — the v1 calibration was single-turn only and left the model badly mismatched on real conversation structure.

How much does quantization cost?

Measured against the unquantized teacher on identical questions (likelihood-scored MMLU, n=200, paired):

Paired: 42 questions the teacher answers and this build misses, versus 8 the other way. McNemar p = 3.06e-06.

This is an honest statement of what 2/3-bit costs on knowledge recall. It is not a reason to avoid the model — it is a 304B model running on a single Apple Silicon machine at 2.91 BPW, and GSM8K (93.5%) and HumanEval (87.2%) remain close to ceiling. Knowledge-recall breadth is where the compression is paid for.

How this recipe compares (all measured, same 200 questions)

Every number below is likelihood-scored MMLU on the identical stratified sample, scored by the same code, paired question-by-question. Absolute values are not interchangeable with the generation-scored table above; the comparisons are what matter.

1. Quantization costs ~17 MMLU points, and that gap is real

Against the native teacher on identical questions: 42 questions the teacher answers that this build misses, versus 8 the other way. McNemar p=3.06e-06. This is an honest statement of what 2/3-bit costs on knowledge recall — and it is unclaimed headroom, not something any method below recovered.

2. AWQ calibration is load-bearing, not a refinement

A plain round-to-nearest build at the same bits and group sizes scores 28.5% — barely above MMLU's 25% chance floor — and generates literal gibberish ('}<?iger}<?codeline...'). Weight reconstruction error was verified at 44.5% RMS, exactly what honest 2-bit RTN produces, so this is not a broken build: at 2-bit, without activation-aware scaling, the model simply collapses. Calibration is the difference between a working model and noise.

3. DWQ distillation added nothing

Distilling the quantized student toward the native teacher (100 steps, expert scales only, agentic-weighted calibration) cut held-out KL by ~89–91% and moved MMLU not at all (p=0.65; near-symmetric 24 gained / 20 lost — churn, not learning). Two loss variants came out statistically identical (p=0.75).

A cautionary result worth repeating: KL-to-teacher over-credits badly. A 91% improvement in the distillation objective bought zero measurable capability. Judge distillation on exact-answer or behavioural evals, never on the loss.

Method note

Teacher and low-bit builds were measured with a streaming likelihood scorer (reap_stream/mmlu_streamed.py) that reads one block at a time — ~8 GB resident — so a 155 GB model can be benchmarked on a 128 GB machine. Prompting is oMLX's own 5-shot MMLU format and stratified sample.

Notes

• oMLX-specific. Uses oMLX's custom deepseek_v4 loader; not portable to generic MLX/GGUF runtimes as-is.
• MTP/DSpark drafter is quantized to the same 2-bit gs128 / 3-bit gs64 recipe (7.04 GB, saving 3.82 GB over native). Safe to quantize aggressively: DSpark uses exact rejection sampling, so every drafted token is verified by the main model — drafter error costs accept-rate (speed) only, never output correctness.
• Native tool calling is DSML (｜DSML｜ markup), not the generic <tool_call> convention.

Sibling model

• DeepSeek-V4-Flash-0731-DWQ — distillation-trained variant of this build. See its card before using it: it shows no measured capability gain over this model and has a suspected early-turn-termination issue.