EAGLE3 Draft Head — Qwen3-Coder-Next

A lightweight EAGLE3 draft head for Qwen3-Coder-Next (80B MoE, 512 experts, 10 active per token, GDN+attention hybrid, 48 layers). Trained with SpecForge on 8x H200 GPUs using the EAGLE-3 training-time test objective.

Qwen3-Coder-Next uses a hybrid layer design that interleaves standard multi-head attention with GDN (linear recurrence) layers. Only 12 of 48 layers are attention layers (every 4th: 3, 7, 11, ..., 47). EAGLE3 auxiliary layers must be selected from attention layers only — GDN layers produce recurrent hidden states that are not compatible with EAGLE3. The model code handles this automatically, selecting layers 3, 23, 47 (first, middle, last attention layers).

Blog post: [TODO: link after publication]

Usage

SGLang (GPU)

Requires our SGLang fork for Qwen3-Coder-Next Eagle3 support.

B=1 server (wide tree — optimal for single-user, real-time requests):

pip install 'git+https://github.com/tails-mpt/sglang.git#subdirectory=python'

python -m sglang.launch_server \
    --model-path Qwen/Qwen3-Coder-Next \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path thoughtworks/Qwen3-Coder-Next-Eagle3 \
    --speculative-num-steps 3 \
    --speculative-num-draft-tokens 8 \
    --speculative-eagle-topk 4 \
    --tp 4 \
    --trust-remote-code \
    --attention-backend triton \
    --port 30000

B=32 server (narrow tree — eliminates Terminal-Bench regression):

python -m sglang.launch_server \
    --model-path Qwen/Qwen3-Coder-Next \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path thoughtworks/Qwen3-Coder-Next-Eagle3 \
    --speculative-num-steps 5 \
    --speculative-num-draft-tokens 6 \
    --speculative-eagle-topk 1 \
    --tp 4 \
    --trust-remote-code \
    --attention-backend triton \
    --port 30002

Important: Wide tree (topk=4) maximizes MT-Bench at B=32 (1.31x) but regresses Terminal-Bench (0.89x). Narrow tree (topk=1) eliminates the regression at the cost of lower peak speedup (1.10x MT-Bench). Use narrow tree for mixed or unknown workloads.

Python Client

import requests

response = requests.post(
    "http://localhost:30000/v1/chat/completions",
    json={
        "model": "default",
        "messages": [{"role": "user", "content": "Write a Python function to merge two sorted lists."}],
        "max_tokens": 512,
        "temperature": 0,
    }
)
print(response.json()["choices"][0]["message"]["content"])

Training Details

Training Method

EAGLE3 trains a single-layer draft head that predicts the next token using hidden states captured from three auxiliary layers of the target model (layers 3, 23, 47 — first, middle, and last attention layers out of 12 total). The training objective is the Training-Time Test (TTT) loss, which simulates the speculative decoding accept/reject process during training to maximize the expected number of accepted tokens at inference time.

GDN (linear recurrence) layers are excluded from auxiliary layer selection because their hidden states encode sequential recurrence rather than per-token representations, making them incompatible with EAGLE3's draft prediction.

Performance

B=1 Inference Benchmarks (temp=0, TP=4, Triton backend)

B=32 Inference Benchmarks (temp=0, TP=4, wide tree)

B=32 Inference Benchmarks (temp=0, TP=4, narrow tree)

Config: B=1 uses steps=3, topk=4, draft_tokens=8. B=32 narrow uses steps=5, topk=1, draft_tokens=6. Hardware: 4x H200 (TP=4), Triton backend. SGLang commit 63291f7f51.

Model Architecture

Limitations

• TP=4 required. FP8 block constraint: shared_expert dim=512, 512/8=64 not divisible by block_n=128.
• Triton attention backend required. FlashInfer is incompatible with head_dim=256 hybrid attention+GDN layers. Pass --attention-backend triton.
• GDN layer constraint. EAGLE3 auxiliary layers must be attention layers (every 4th), not GDN layers. The model code handles this automatically.
• Temperature sensitivity. Best performance at temp=0 (greedy). MoE expert routing is non-deterministic at temp>0, which reduces draft acceptance rates.
• Terminal-Bench regression at B=32. Wide tree (topk=4) regresses Terminal-Bench to 0.89x. Use narrow tree (topk=1) for mixed workloads.
• Requires SGLang fork. Upstream SGLang does not yet include the Qwen3-Next EAGLE3 patches.

License

This draft head is released under Apache 2.0, matching the Qwen3-Coder-Next license.

Citation

@inproceedings{li2025eagle3,
  title={{EAGLE-3}: Scaling up Inference Acceleration of Large Language Models via Training-Time Test},
  author={Li, Yuhui and Wei, Fangyun and Zhang, Chao and Zhang, Hongyang},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
  year={2025}
}