AEON DFlash — Qwen3.6 / Ornith 35B-A3B
A DFlash speculative-decoding drafter for Qwen/Qwen3.6-35B-A3B and its derivatives — including Ornith-1.0-35B-AEON-Ultimate-Uncensored (a light RL post-train of that base). ~0.9 GB. Gives ~1.9× single-stream decode on a DGX Spark with no quality loss (lossless speculative decoding: the target model verifies every token).
What makes it different
This drafter is 8 layers of full attention — deliberately no sliding-window attention (SWA). That single design choice is the difference:
| This drafter (AEON) | Typical SWA drafter (e.g. z-lab) | |
|---|---|---|
| Layers | 8 × full attention | 6 × (5 sliding + 1 full), window 4096 |
| KV page size | aligns with the target's pages | larger (windowed) → mismatch |
| Serves on the AEON vLLM image | yes, no patch | needs a kv_cache_utils patch |
Why the no-patch part matters. Qwen3.6-35B-A3B / Ornith is a hybrid stack — full attention + GatedDeltaNet (Mamba) recurrent state. vLLM unifies KV-cache page sizes across layer types by scaling block_size, but Mamba/GDN state pages are shape-based and don't scale. A drafter with SWA layers introduces a larger KV page that can't be reconciled with the GDN pages → vLLM aborts at engine init with unify_kv_cache_spec_page_size … AssertionError (see issue #1). An all-full-attention drafter has no oversized page, so it loads cleanly on the stock AEON image — a true drop-in.
It also targets the right hidden states for this stack: DFlash reads target layers [1, 10, 19, 28, 37] of the 40-layer 35B-A3B model.
A note on KV cache (and why SWA doesn't help here)
This model serves with BF16 KV cache, and a sliding-window drafter does not change that. The BF16-KV requirement comes from two places, neither of which is the drafter's window type:
- the vision tower (non-causal vision attention → FA2 backend only → no FP8 KV), and
- DFlash's non-causal verification (block-parallel → FA2/FLEX only → no FP8 KV).
KV dtype is a property of the main model's attention backend, not the drafter's layers. SWA changes the drafter's own (tiny, windowed) KV footprint only — it can't unlock FP8 KV for the target.
Usage
Point any Qwen3.6-35B-A3B / Ornith deploy's --speculative-config at this drafter:
--speculative-config '{"method":"dflash","model":"/drafter","num_speculative_tokens":6}'
Validated on a DGX Spark (GB10) with the Ornith NVFP4 QuickStart. ⚠️ Keep --max-num-seqs ≤ 16 with DFlash on the Spark (unified-memory limit — see the QuickStart).
Performance
Head-to-head on Ornith-1.0-35B NVFP4 (DGX Spark GB10 · n=6 · max-num-seqs 16 · single stream):
| Drafter | c=1 decode | Accept length | Draft accept rate | Patch needed |
|---|---|---|---|---|
| AEON (8L full-attn) — this repo | 72.9 tok/s | 3.71 | 45.1% | none |
| z-lab (6L SWA) | 74.4 tok/s | 3.35 | 39.2% | KV page-size hotfix |
Speed is effectively tied (c=1 within ~2% — noise; this drafter is slightly ahead at c=8 aggregate). This drafter wins on acceptance — its 8 full-attention layers draft more accurately (3.71 vs 3.35 tokens accepted per step; 45% vs 39%) than the SWA variant's 6 layers, which trade accuracy for a cheaper forward pass. Combined with no patch required and no padded-Mamba-page overhead, it's the recommended default. Both are lossless (the target verifies every token); ~1.9× over plain decode either way.
Support the work
AEON-7 models, drafters and tools are built and trained independently, on my own hardware. If they're useful to you, please consider supporting development:
Become a member on Patreon → patreon.com/cw/AeonForge7/membership
Milestones unlock bigger work: reaching 500 paid supporters will fund fine-tuning larger models and bigger project releases. Supporters also get early access to new releases, such as the AEON DFlash2 drafter.
Credits
- Drafter: AEON-7. Method: DFlash speculative decoding.
- KV page-size diagnosis + the patch for SWA-drafter compatibility: @newjordan (issue #1 / PR #2).
License: MIT.