GLM-5.3-Flash DFlash2 Drafter — run dflash2E

Superseded (2026-09-24) by canada-quant/GLM-5.3-Flash-DFlash2-F, and on 2026-09-25 by canada-quant/GLM-5.3-Flash-DFlash2-G (current) — same architecture and serve contract (drop-in: change the path, keep num_speculative_tokens: 7). Same 8× B300, same protocol: -G 3.676 / -F 3.626 / this drafter 3.561 mean acceptance at K=7 vs the incoai reference's 3.632. This repo stays up unchanged as the record behind the numbers below and the Spark A/B of 2026-09-15/16.

Self-trained DFlash2 block-diffusion speculative-decoding drafter for GLM-5.3-Flash, trained against our INT4 quant canada-quant/GLM-5.3-Flash-W4A16-MTP. Trained from scratch on self-generated data — no third-party drafter weights or traces anywhere in the training path — and adopted on 2026-09-15 as the authors' serving drafter-of-record on their 2× DGX Spark (SM121) stack, superseding our previous best dflash2DF2 (owner ruling; the swap is verified and the stack serves continuously since).

Architecture

Training

• Warm-started from our previous best dflash2DF2 (L8T9 lineage, third refinement round).
• 350,260 self-generated samples (selfgen_v13), 35,021 steps, lr 1e-4.
• Data: prompts sampled from public instruction sets (ultrachat_200k · MIT, OpenR1-Math-220k · Apache-2.0, OpenCodeReasoning / OpenCodeInstruct · CC-BY-4.0, evol-codealpaca-v1 · Apache-2.0); every completion was regenerated by the target model itself (thinking ON) — no third-party model outputs.
• The widely-used reference drafter incoai/GLM-5.3-Flash-DFlash2 was never a training input — it appears below only as a measured same-protocol comparison.
• PROVENANCE.txt in this repo is the verbatim training record (run name, warm-start source, data, steps, eval numbers).

H200 holdout evaluation

500 never-trained-on prompts, thinking ON, T=1.0 / top_p 0.95, max_tokens 1024, greedy drafts, H200 TP=4 (concurrency 16; the c1 row is 100 prompts at concurrency 1):

Per-position acceptance (K=7, c16): 0.744 · 0.545 · 0.405 · 0.306 · 0.237 · 0.185 · 0.147.

Serving it on 2× DGX Spark (SM121)

Prebuilt vLLM image + one-command launcher: canada-quant/vllm-glm53-flash-sm121 (docker pull ghcr.io/canada-quant/vllm-glm53-flash-sm121:v2-w4a16-dflash2e). The drafter is bind-mounted at runtime — swap drafters without a rebuild; K follows the drafter's block size (E: block 8 → num_speculative_tokens=7, the trained value).

Same-rig A/B — 2× DGX Spark (SM121), TP=2, eager, K=7

All three drafters measured on the same rig within ~10 hours, same 500-prompt holdout family, same W4A16 target:

Honest read:

• Vs our previous best (dflash2DF2): wins every cell — c16 +0.077 acceptance / +2.2% throughput; c1 +0.087 / +2.0%; smoke +0.067. This is why it was adopted.
• Vs the incoai reference: it loses the c16 throughput primary by 2.9% (70.87 vs 72.98 tok/s; acceptance 3.5788 vs 3.6274) and the smoke cell, while edging c1 acceptance (+0.021; 3.5970 vs 3.5760) at −2.9% tok/s. We ship it anyway: it is fully ours (Apache-2.0, no NC/ND terms), trained on data we control, reproducible end-to-end, and it closed 61% of the acceptance gap our previous best had left open.
• Cross-hardware port fidelity: Sparks c16 3.5788 vs H200 3.568 (+0.3%); c1 3.5970 vs 3.585 (+0.3%) — the acceptance profile reproduces across SM121 and H200.

Serving

Pairs with canada-quant/GLM-5.3-Flash-W4A16-MTP (the quant it was trained against) or BF16 GLM-5.3-Flash. vLLM speculative config:

{"method": "dflash", "model": "/models/GLM-5.3-Flash-DFlash2-E", "num_speculative_tokens": 7}

Ready-to-run recipe: launch_dflash2_tp2.sh ships in this repo — the complete TP=2 DGX-Spark serving launcher (defaults to the W4A16 target + this drafter; GRAPHS/EAGER knobs, DRAFTER_HOST_PATH override to serve any other DFlash2 drafter).

Hard constraints (all measured, not stylistic):

• num_speculative_tokens: 7 (= block_size − 1) is the trained and measured value. k=5 also boots on the SM121 image (2026-09-26, not benchmarked); an earlier version of this card said other counts boot-wedge the stack, which was wrong.
• mask_embedding.pt must sit next to the weights. Verify the boot log carries Loaded DFlash mask embedding for mask_token_id 154856 from mask_embedding.pt — absence means the mask was silently ignored; do not serve.
• The 9-tap config requires the serving stack to honor dflash_config.target_layer_ids of length 9 (upstream vLLM DFlash2 does — vllm-project/vllm#52816).
• Full-attention drafter layers: the target stack must accept FullAttentionSpec drafter KV in the GLM-5 KV fast path. The stock upstream DFlash2 image cannot boot this drafter; see the W4A16 card's SM121 serving section for the working stack (patched SM121 build, fabric tuning, launcher).

Our serving configuration: 2× DGX Spark GB10, TP=2 + expert-parallel, 262K context, fp8_e4m3 KV cache, K=7 greedy drafts — left up and serving since 2026-09-15 (smoke 4.243 @ 31.2 tok/s, c1 3.617 @ 28.2 tok/s post-swap verification, zero stream errors). Serving state (updated 2026-09-16): the stack now runs the ruled graphs-ON g4 config — CUDA-graphs FULL_AND_PIECEWISE capture [1,2,4,8,16,24,32] + VLLM_USE_BREAKABLE_CUDAGRAPH=0, GMU=0.795, KV 8 GiB (pool 366,749 tokens); the flip boot re-verified all five production gates (pool discriminator, graphs live on both ranks, mask-loader line, zero-error smoke, zero engine faults). The DFlash2 speculator + selector walk the vLLM V2 GPU-worker path at serve time (confirmed 2026-09-16 from the boot bytes) — no V1-proposer silent-skip.

Memory cost — and when to use the built-in MTP head instead (added 2026-10-04)

This drafter's 8 layers use full attention, so each keeps a KV cache for the whole context (same architecture as -F / -G); the incoai reference drafter keeps a 2,048-token sliding window. KV pools measured by the authors on 2× DGX Spark (fp8 KV):

That is about 3× less context per GiB of KV. A 9 GiB pin holds ≈450K tokens with -E / -F / -G — less than one 1M request — so 1M context on 2× Spark is validated only with the incoai drafter; with our drafters the validated contexts are 262K (8 GiB) and 800K (16 GiB).

• Use this drafter for short-context English and code, where its acceptance is highest.
• Use GLM-5.3-Flash's built-in BF16 MTP head (shipped in the W4A16 checkpoint; {"method":"mtp","num_speculative_tokens":3}) for long context, non-English text, or KV headroom.
• Measured by the authors (W4A16 + -G, 2× Spark, our SM121 image, single stream, llama-benchy pp2048/tg128): 31.4 / 30.7 / 15.3 / 10.0 tok/s at context depth 0 / 4K / 65K / 100K (2026-09-28; the 100K cell from 2026-09-26).
• Community-reported (NVIDIA developer forum, 2026-10-04; not measured by us): on eugr's b12x vLLM build with the GPU clock capped at 1700 MHz, the W4A16 weights with the MTP head held 31.6 / 29.8 / 25.4 / 31.0 tok/s at the same depths; with -G usable context fell from ≈930K to ≈330K tokens and decode was 20–30% slower on non-English text (faster on English code).
• Our own same-pair comparison on 2× Spark (drafter vs MTP, plus eugr's b12x build) has been running since 2026-10-04 on a bilingual generation workload; results will be published in canada-quant/vllm-glm53-flash-sm121.

Provenance

• Files: model.safetensors (sha256 af0d8d16f01f…), config.json (699bdf29…), mask_embedding.pt (2474994c…), PROVENANCE.txt (verbatim record).
• Every number on this card comes from sha256-pinned banked artifacts (raw eval JSONs, byte-verified measurement journals); the A/B table is the adjudicated table of record from the measurement program.

References

• DFlash2 method: z-lab/dflash · arXiv 2602.06036 · inco.ai blog
• Upstream vLLM DFlash2 integration: vllm-project/vllm#52816
• Target model: zai-org/GLM-5.3-Flash · our quant: canada-quant/GLM-5.3-Flash-W4A16-MTP
• Reference drafter (comparison only): incoai/GLM-5.3-Flash-DFlash2 (CC-BY-NC-ND-4.0)

This card was assembled from sha-pinned banked measurement artifacts; reviewed by a human (the maintainer) before publication.