DeepSeek-V4-Flash-0731 — MXFP4 (MLX)

MXFP4 experts + MXFP8 attention, in MLX format. This is a bit-exact conversion of deepseek-ai/DeepSeek-V4-Flash-0731 (305B params, 13B active). Every weight dequantizes to exactly the same value as the original checkpoint — verified maxdiff = 0.000e+00 — at +0.12% size.

[!IMPORTANT] This checkpoint keeps the DSpark MTP heads. Most community builds strip them.

oMLX 0.5.4rc2+ ships native DSpark Lightning MTP with Metal kernels for the DSpark projections, routed experts, DSA indexer scoring and block verification. It is inert without the embedded dspark_* weights. Several MLX conversions drop them — quantization pipelines commonly strip auxiliary heads, since they are dead weight unless the runtime knows how to use them. Check for mtp.* tensors in any build before relying on native MTP.

On this checkpoint, native MTP measures 41.7 tok/s at 23K context versus 26.1 tok/s with speculation off — on the same machine, same harness. That difference is the whole reason to prefer this build.

Set mtp_enabled for the model (oMLX stores it in ~/.omlx/model_settings.json) and you are done.

[!WARNING] On oMLX 0.5.4rc2+, do not install the companion speed patches. The patch repo predates native MTP. Its speculation marker puts a second speculative decoder on the same KV cache — output stays fluent while repeatedly restarting the same sentence — and its cache patches abort rc2 with Cache corruption not recoverable after retries. Upstream is also faster at long context. The repo is kept as a record of how this checkpoint and its measurements were produced.

[!TIP] Check the prompt cache before anything else. Under oMLX, hot_cache_max_size has shipped at "0" (disabled). Enabling it took a repeated 23.7K-token prompt from 51.0s to 4.6s — an 11x improvement worth more than every kernel change in this project combined. It lives under Memory Management, not the CACHE panel.

Measured performance

Single Mac Studio, M3 Ultra (80-core GPU, 256 GB unified memory, 819 GB/s), macOS 26.6, mlx 0.32.0 / mlx-lm 0.31.3 under oMLX. Weights resident: 145.5 GiB.

Decode

Measured by slope (generate N1 and N2 tokens from the same prompt, take (N2-N1)/(t2-t1)), which cancels prefill. M3 Ultra, 23K cached context.

Recommended: stock oMLX 0.5.4rc2 with mtp_enabled. No patches, no markers.

For the record, the pre-rc2 patch stack on 0.5.4rc1 measured 37.6–40.7 at 23K and 47.7–50.1 short. Ranges, not points: the same configuration measured at the top of those ranges shortly after a reboot and at the bottom after sustained load, with settings verified identical. Machine state moves these numbers by ~8% on this hardware; quote the low end of anything you rely on.

Published third-party figures for this model on the same hardware, all without MTP (their checkpoints strip it): 26.2 tok/s (oMLX's own benchmark), 28.8–29.8 warm (al-engr), 35.5 short / 26.6 at 12K (community Q4).

For comparison: dual DGX Spark

A published dual-Spark run of this model reports 72.8 tok/s single-stream — faster, on two nodes with TP=2 halving per-node weight traffic. At concurrency the picture changes:

Per-stream drops to 35.4 at x4, so a single M3 Ultra is level or ahead once there is any concurrency — and its cached prefill at 23K is ~1.2s.

Earlier revisions of this card quoted 47.1, 2.06x, 43-48 and 938 tok/s. All withdrawn — variously measured against a corrupt KV cache, by a prefill-contaminated method, or on pathologically repetitive text. The patch repo documents each.

If you run this under oMLX, check the prompt cache first

oMLX ships hot_cache_max_size at "0", which means disabled. With "Hot Cache Only" set (RAM, no SSD spill) nothing is cached and every request re-prefills its whole prompt. The admin UI's CACHE panel does not show the size field — it is under Memory Management → "Memory Limit (In-Memory Hot Cache)".

On a repeated 23.7K-token prompt, setting it: 51.0s → 4.6s. For agent traffic that resends a growing context each turn this matters far more than any decode tuning.

Output head

The checkpoint leaves head unquantized, so it is 129280x4096 bf16 = 1.06 GB read per forward. Quantizing to 8-bit halves that, measured teacher-forced over 2047 positions:

8-bit is effectively lossless. 4-bit is clearly degraded.

DSpark acceptance

Measured against ground-truth autoregressive decoding, no rollback involved, at the drafter's trained width on clean code:

Strongly content-dependent. In a real generation loop against mixed chat traffic it runs nearer 2.0/3, which is what the decode table reflects.

Prefill

A previously published 938 tok/s figure is withdrawn — it was measured on one three-line function repeated 800 times, which collapses MoE routing. Real content measures ~430 tok/s at 25K.

Attention builds a dense (L, L) mask despite a 128-token sliding window, so prefill carries an O(L^2) term. The blocked/windowed path in the patch repo removes it. The A/B below was taken on random tokens — worst case for the dense path — so read the shape rather than the ratio:

The dense path halves from 8K to 16K; the windowed path does not move. The equivalent comparison on real text has not been measured.

Reference point

The 2x DGX Spark recipe (tonyd2wild/DeepSeek-v4-Flash-0731-DSpark-1M-NVFP4-KV-2x-DGX-Spark) reports 84.3 tok/s peak decode on structured prompts and ~22 tok/s per stream on mixed agent traffic, across two TP-sharded nodes. The numbers above are one Mac Studio.

What makes this build different

The MTP heads are present. mtp.* tensors are included, so oMLX 0.5.4rc2+ can drive native DSpark Lightning MTP against this checkpoint. That is the difference between 41.7 and 26.1 tok/s at 23K context on an M3 Ultra. If you want to verify before downloading 167 GB, look for mtp. keys in model.safetensors.index.json.

The experts are a byte transplant, not a requantization. DeepSeek shipped this model already quantized (166.9 GB for 305B parameters — about 4.38 bits per weight). This repo re-containers those MXFP4 bytes into MLX's format without touching the values: every weight dequantizes to exactly the original number, verified maxdiff = 0.000e+00. Requantization error on the routed experts — 98% of the parameters — is therefore zero by construction rather than small.

Any conversion that dequantizes and re-quantizes adds error on top of DeepSeek's own quantization. How much depends entirely on the scheme and is worth measuring rather than assuming.

Perplexity is not yet published for this repo. The bit-exactness argument above is a statement about the weights, not a measured quality claim, and should not be read as one until a wikitext-2 number with a stated protocol appears here.

Loading

[!IMPORTANT] Earlier revisions of this repo did not load. config.json keyed its quantization overrides by checkpoint tensor name (layers.0.attn.wq_a), but MLX's nn.quantize passes class_predicate a module path (model.layers.0.attn.wq_a) — sanitize renames tensors on the way in and nothing applied the same renaming to the quantization keys. Every lookup missed and all 390 MXFP8 attention/shared-expert modules silently fell back to the MXFP4 default, failing with:

ValueError: Expected shape (1024, 512) but received shape (1024, 1024)
             for parameter model.layers.0.attn.wq_a.weight

(4096/8 = 512 lanes at 4 bits vs the true 4096/4 = 1024 at 8 bits.)

Fixed as of this revision. config.json now carries both spellings, so it resolves under either convention. If you pinned an older revision, re-pull it.

You still need a deepseek_v4 modeling module — stock mlx-lm has none (#1233, #1281). oMLX ships one.

Note on the DSpark heads

config.json says num_nextn_predict_layers: 1, but the checkpoint ships three heterogeneous DSpark stages under mtp.* (inference/config.json has it right as n_mtp_layers: 3):

• mtp.0 — main_proj (3*dim -> dim, fusing hidden states from layers 40/41/42 per dspark_target_layer_ids) + main_norm, then a block
• mtp.1 — block only
• mtp.2 — block + norm + hc_head + markov_head (low-rank bigram prior)
  • confidence_head

This is not the DeepSeek-V3 MTP head (e_proj/h_proj, enorm/hnorm), so V3-shaped MTP implementations will not load it. The field is deliberately left at 1: raising it to 3 makes such implementations build three V3-shaped heads and fail. Speculative decoding needs a DSpark-aware implementation.

Why this is lossless

The source is already stored in formats MLX supports natively — it just isn't labelled that way:

DeepSeek's expert format is OCP MXFP4. So rather than dequantizing to bfloat16 and re-quantizing — which loses precision and inflates size — this conversion transplants the payload bytes directly into MLX's layout:

• MXFP4 — 4 source bytes (8 e2m1 nibbles) reinterpret as one uint32 lane; the e8m0 scales already have exactly the per-row, per-32 granularity MLX wants, so they pass through untouched. Zero transformation.
• MXFP8 — 4 e4m3 bytes per uint32 lane; the one difference is that MLX keys scales per 32 values while the source keys them per 128×128 block, so each block scale is broadcast to its 4 groups. Same values, finer indexing.

The only size change in the whole model is that MXFP8 broadcast: 1 byte per 32 values instead of 1 byte per 16,384. That is the entire +0.12%.

Note mx.quantize(mode="mxfp4") is not bit-exact here (~2.2% error) — it re-derives its own scale exponent from bfloat16 rather than reusing DeepSeek's. The byte transplant sidesteps that.

Verification

• e4m3 and e8m0 decode tables match ml_dtypes exactly on all 256 codes.
• Both dequant paths reproduce an independent NumPy reference bit-for-bit.
• 200 randomly sampled modules across both modes: mx.dequantize of this repo equals the source dequantization exactly, zero mismatches.
• Full tensor accounting: every source module and passthrough tensor present.
• Quantization metadata independently re-derived from the safetensors headers via the packing ratio (scales_last == lanes/4 -> mxfp4, lanes/8 -> mxfp8): 390 MXFP8 + 35,328 MXFP4, zero mismatches, zero stray keys.

Layout

Source tensor naming is preserved (layers.N.attn.wq_a, layers.N.ffn.experts.E.w1, embed, head). Each quantized module emits {mod}.weight (uint32) and {mod}.scales (uint8, e8m0) — no biases, since MXFP4/MXFP8 use pure power-of-2 scales. Experts are stored per-expert, not stacked.

config.json carries quantization with mxfp4 as the default (the ~35k expert modules) and explicit per-module overrides for the mxfp8 and unquantized tensors, in both checkpoint-tensor and MLX-module-path spellings.

License

MIT, inherited from the base model.