EAGLE-3 draft head for Nemotron 3.5 Lightning

An EAGLE-3 draft head for NVIDIA Nemotron 3.5 Lightning, a hybrid Mamba-2, attention and mixture-of-experts model of approximately 30B total and 3B active parameters. The head is one transformer layer of 400 MB, trained with SpecForge and a nemotron_h backend added for this target, on 8 × B200 at TP=2 using the EAGLE-3 training-time test objective, and benchmarked on H200 and B200.

The EAGLE-3 write-up supplied with this card, EAGLE-3 speculative decoding for NVIDIA Nemotron 3.5 Lightning, is the full measurement record.

Which checkpoint this head is for

This head drafts for the base checkpoint of the family, nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Base-BF16, and it was trained against those weights.

The study behind it measures the raw model: what native multi-token prediction is worth exactly as the base weights ship, and how a purpose-trained EAGLE-3 head compares against it on those same weights. The base checkpoint is what keeps that a measurement of the model itself, with no post-training recipe layered on top of it.

NVIDIA publishes other checkpoints in this family, including a post-trained variant and the NVFP4 DFlash and DSpark speculative checkpoints. This head was not trained against any of them and was not measured against any of them. Pairing it with one may still load and run, but nothing in this card predicts what it would do, and none of the throughput or acceptance figures below would apply.

Contents

1. Comparison baseline
2. Target model requirement
3. Serving configuration
4. Determinism of the serving stack
5. Measured results
6. Output equivalence
7. Router integration
8. Architecture
9. Training
10. Limitations
11. License
12. Citation

1. Comparison baseline

Nemotron 3.5 Lightning ships its own native multi-token prediction head. Every figure in this card is measured against native MTP on the same node, the same engine, the same dataset and the same draft budget. Figures against unaccelerated decoding are given alongside for scale.

At equal draft budget on H200 with vLLM, at batch size 1 and temperature 0, over twenty repeats per cell, this head matches native MTP throughput on mathematics, exceeds it by 1% on code, and falls below it by 11% on general chat. Task accuracy is equivalent to unaccelerated decoding: GSM8K exact-match at temperature 0 under concurrency 32 is 0.6869 against 0.6823, and equivalence within ±0.05 is established by two one-sided tests at p < 0.01, n=1319. Section 5 gives the intervals and the sample sizes, and Section 10 states what the evidence does not cover, including why byte-exact agreement is not the instrument used here.

2. Target model requirement

This repository contains the draft head alone, 400 MB in two files. Serving it requires nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Base-BF16, obtained separately, and nothing in this repository functions without it.

The head was trained and measured against the NVIDIA early-access build of this target, dated 6 July 2026, which was the only form available while the work was carried out. No published checkpoint is byte-identical to it: none of that build's fourteen shard digests match the released weights. Numbers in this card were produced on the early-access build and have not been re-measured against the public release.

The head ships config.json and model.safetensors. It carries no tokenizer files, which is expected: both engines load the tokenizer from the target path, so an EAGLE-3 draft directory does not require one of its own.

3. Serving configuration

3.1 vLLM

pip install vllm==0.25.1

VLLM_ATTENTION_BACKEND=FLASHINFER \
CUDA_VISIBLE_DEVICES=0 vllm serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Base-BF16 \
  --trust-remote-code --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.85 --max-model-len 8192 \
  --mamba-backend flashinfer --moe-backend flashinfer_cutlass --no-enable-prefix-caching \
  --speculative-config '{"model":"thoughtworks/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Eagle3","method":"eagle3","num_speculative_tokens":3}' \
  --served-model-name nemotron35 --port 30032

The head serves on unmodified upstream vLLM, with no fork, no conversion step and no patch. It is authored as a one-layer Llama: config.json declares architectures: ["LlamaForCausalLMEagle3"] and model_type: "llama", and upstream vLLM supplies the matching registry entry, the weight-key remapper for SpecForge's layout, and target-side auxiliary hidden-state capture for nemotron_h through SupportsEagle3.

vLLM is the faster of the two engines on this model, by 1.32× on unaccelerated decoding and by 1.51× with this head, measured over 63 paired cells against SGLang on one node.

3.2 SGLang

# Apply the nemotron_h auxiliary-capture patch described below to SGLang first, then:

CUDA_VISIBLE_DEVICES=0 python -m sglang.launch_server \
  --model-path nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Base-BF16 --trust-remote-code --tp 1 \
  --speculative-algorithm EAGLE3 \
  --speculative-draft-model-path thoughtworks/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Eagle3 \
  --speculative-num-steps 5 --speculative-eagle-topk 1 --speculative-num-draft-tokens 6 \
  --mem-fraction-static 0.75 --mamba-ssm-dtype float32 --max-mamba-cache-size 256 \
  --disable-radix-cache --enable-metrics \
  --cuda-graph-max-bs 64 --max-running-requests 64 --port 30010

Upstream SGLang cannot serve this head, having no auxiliary hidden-state capture for nemotron_h. The Thoughtworks fork adds it in one commit, one file and +57/−4 lines. That repository is not public; the patch is supplied on request and the commit is offered upstream.

Two requirements apply to this configuration. ninja must be present on PATH as an executable, because FlashInfer's just-in-time compilation shells out to it; a missing ninja terminates the server approximately four minutes after launch, once weights have loaded, which presents as a model fault. --disable-radix-cache is required for nemotron_h speculative decoding.

3.3 Client

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:30032/v1", api_key="none")
r = client.chat.completions.create(
    model="nemotron35",
    messages=[{"role": "user", "content": "Explain speculative decoding in two sentences."}],
    temperature=0,
)
print(r.choices[0].message.content)

The endpoint is an ordinary OpenAI-compatible chat endpoint. Speculative decoding is not visible to the client: the request, the response schema and the model id are unchanged by it.

4. Determinism of the serving stack

The configuration in Section 3.1 is the throughput-optimal one and is also non-deterministic. Under --moe-backend flashinfer_cutlass this model is not reproducible run to run at temperature 0: the same server given the same prompt twice returns identical text approximately 20% of the time, and 28% with that flag applied alone. --mamba-backend flashinfer is deterministic in isolation. The likely mechanism is a non-associative reduction in the fused MoE router, which routes each token to 6 of 128 experts.

Two consequences follow. The equivalence result in Section 6 was measured on the deterministic default backends, because on the throughput-optimal configuration the engine's own self-agreement rate of approximately 20% exceeds any drafter effect; an exact-match test run on those flags returns 6% to 22% by dataset, against the same server's 20% agreement with itself, and is indistinguishable from the result a broken head would produce. Separately, outputs are not batch-invariant on any backend: a server that is 100% self-consistent at batch 1 agrees with itself approximately 31% of the time at concurrency 32. That property belongs to the serving stack and applies equally to native MTP and to unaccelerated decoding.

5. Measured results

All measurements are against native MTP on the same node, engine, dataset and draft budget.

5.1 Statistically qualified result

vLLM, H200, batch size 1, temperature 0, natural stopping, equal draft budget of 3 tokens for both arms, twenty repeated measurements per arm per domain within one server session per arm, both arms measured in the same twenty windows on the same node, 4096-token cap on whole problems. The intervals are two-sample delta-method intervals at n = 20 per arm and do not cover boot-to-boot variation.

Mathematics is at parity: the interval spans 1.0. Code is above it and chat is below it, both outside their intervals.

These figures supersede an earlier set measured at a 512-token cap on the fragment corpus, which read +4.76% and +2.65% over four server boots, and −5.76% from twenty repeats within one boot. Two things changed together. The cap rose from 512 to 4096, and the code corpus moved from newline-split fragments to whole problems. At 512 the corrected corpus truncates many generations before the model stops, so the earlier code and mathematics figures describe a partly forced-length regime; the chat figure moves furthest because chat generations are longest.

5.2 Domain and hardware dependence

Both columns come from the Section 5.1 lane: H200, batch size 1, an equal three-token draft budget for both arms, twenty repeated measurements per arm. A figure measured at each arm's own best draft budget differs; the write-up's Section 4.1 table reports that lane separately.

The Section 5.1 code figures use the corrected 164-problem HumanEval corpus. The write-up's single-stream table, and every figure derived from it, uses an earlier corpus split on newlines rather than on problem boundaries, whose records are single lines of code or docstring. Both arms received identical prompts in the same session, so the ratio is a controlled measurement; the workload is narrower than the domain label indicates. Every other code figure in this card comes from that earlier corpus, and re-measurement of those cells on the 164-problem corpus is outstanding.

Results by configuration:

• H200, vLLM, batch 1, equal budget: the result in Section 5.1.
• H200 under concurrency: at or slightly above parity on mathematics and code from 1 to 64 concurrent requests.
• B200, single stream: 0.77 to 0.85× at equal draft budget, and 0.80 to 0.86× at each method's best configuration. Native MTP's cheaper draft wins at batch 1 on B200, and the deficit closes by 4 concurrent requests on mathematics and code; on chat it does not close at any concurrency measured.
• SGLang on B200 with tree drafting: above native MTP on all three domains, by 19%, 17% and 2%, and the highest acceptance measured in the study at τ 4.95. Single measurement, on the slower engine.
• Fixed-length concurrency ladder, output forced to 512 tokens: native MTP leads on every domain at 128 concurrent requests, because a 3-token draft continues to scale where a 5-token budget saturates the verification step. The forced length of 512 differs from the 1024 of the canonical protocol, so those figures are not comparable to fixed-length results measured at 1024.

The H200 and B200 figures were measured on two different draft heads, which the study's Section 8 item 5 identifies as heads A and C. The head published here is head A, and every repeated measurement was taken on it. No lane measures both heads on one machine across the full matrix.

5.3 Acceptance behaviour

Acceptance, written τ, is the mean number of tokens accepted per verification step. It moves by less than 2% from 1 to 64 concurrent requests on mathematics and code, while the speedup against unaccelerated decoding falls from 1.67× to 1.42×. Acceptance holds over that range while the GPU runs out of spare compute to spend on verification. The serving implication is that speculation should follow batch size rather than an observed acceptance drop, because acceptance does not signal the transition.

At equal draft budget, acceptance is below native MTP's in all six batch-1 cells, at 0.85 to 0.95×. Where this head wins throughput it does so with fewer predicted tokens per step and a cheaper draft forward pass. That is a hardware-cost effect, and an improvement to the draft kernels would reduce or remove it.

Higher acceptance can yield lower throughput. On chat the 5-token chain reaches τ 2.488 against the 3-token chain's 2.281, and 244.7 tok/s against 259.7. This follows the closed form speedup ≈ τ / (N·c + 1), in which N is the draft budget and c the cost of one draft forward pass as a fraction of one target forward pass: the deeper chain buys acceptance and buys more cost than acceptance. Acceptance is a driver of throughput rather than a criterion for selecting a configuration.

6. Output equivalence

Speculative decoding is lossless at greedy sampling by construction, because each drafted token is verified against the target's own distribution. At greedy the target's distribution is a point mass, so in exact arithmetic that guarantee implies identical tokens, and byte-exact agreement is a valid instrument for it. Finite precision limits that instrument to cases where the verification path and the sequential path resolve near-ties identically, and on this model they diverge.

Task accuracy is the instrument used here. GSM8K exact-match, temperature 0, concurrency 32, n=1319 problems per arm, on H200 with vLLM 0.25.1:

Equivalence within ±0.05 is established by two one-sided tests, both rejected at p < 0.01. A single hypothesis test would show only a failure to reject equality, which is a weaker statement. The margin is bounded by the sample rather than by the difference: at n=1319 the standard error of the difference is 0.018, so no margin below ±0.030 is demonstrable however small the true gap, and this arm's observed difference supports ±0.035.

Byte-exact agreement was measured in July 2026 at a 512-token cap on a fragment corpus and returned 150 of 150. It does not reproduce at a 4096-token cap on whole problems, and the reason belongs to the serving stack rather than to any draft head. Two servers started from one flag set either agree almost completely or disagree almost completely, and which of the two is fixed when the process starts. Between two servers first shown to be in the same launch state, an unaccelerated pair agrees on 164 of 164, and both the model's own native MTP head and this head agree with unaccelerated decoding on 153 of 164 on code, 163 of 164 on mathematics and 154 of 154 on chat, at the same prompt indices and in the same counts. Two servers that land in different states agree on as few as 4 of 164, and that effect reproduces with no draft head present on either side. A separate lane measured the first divergence of each disagreeing pair at a 512-token cap over 60 prompts per cell: across 132 divergences no token fell outside the unaccelerated model's top-5 candidates, every logprob gap was a multiple of 0.125, the bf16 spacing in this range, the median gap was two of those steps and 27% were exact ties.

The mechanism is that the verification pass and the sequential pass reach the same near-tied argmax through different kernels and resolve it differently, after which the sequences separate. Deployments that require byte-reproducible output should pin the serving configuration and verify it on their own corpus; deployments that require task accuracy have the evidence above.

At temperature 1.0 equivalence also holds. Over twenty runs per speculative arm and eighteen for unaccelerated, GSM8K exact-match is 0.6834 unaccelerated, 0.6801 native MTP and 0.6817 for this head, and two one-sided tests at ±0.05 reject both nulls at p < 0.0001. The unit of analysis is the run rather than the problem, because sampled decoding varies run to run.

7. Router integration

Three properties govern deployment behind a model router or gateway.

Speculative decoding is server-side configuration on the worker, so the head is invisible to the router. The router observes the same OpenAI-compatible endpoint, the same model id and the same response schema whether speculation is enabled or not, and no integration work is required beyond directing traffic at a worker started with the flags in Section 3.1.

Routing to a speculative worker preserves task accuracy: Section 6 gives equivalence within ±0.05 on GSM8K at temperature 0 under concurrency, established by two one-sided tests at p < 0.01. It does not preserve the exact token sequence. At a 4096-token cap on whole problems, between two servers in the same launch state, the speculative and unaccelerated arms differ on 11 of 164 code prompts and 1 of 164 mathematics prompts, and the model's own native MTP head differs on the same prompts in the same counts. Two servers in different launch states differ on most prompts whether or not speculation is enabled, for the reason Section 6 gives. On the throughput-optimal configuration the engine is additionally non-deterministic run to run, so two calls may differ whether or not speculation is enabled.

An integration that depends on task accuracy can substitute a speculative worker freely. An integration that depends on reproducing a specific string, a cached response keyed on output, a golden-file test, or a diff against a previous run, should pin the serving configuration and verify on its own corpus before substituting.

The throughput result is domain-dependent and batch-dependent, which makes both a routing input. Mathematics and code gain throughput on H200 at low and moderate concurrency. General chat gains only on SGLang with tree drafting on B200, by 2%, and falls below native MTP on every vLLM configuration measured. At 128 concurrent requests under forced output length, native MTP leads on every domain.

8. Architecture

The key-value head count is a hard constraint rather than a tuning choice. Earlier heads for this target used 7 key-value heads against the target's 2, and vLLM could not reconcile the KV-cache page size against a Mamba-2 target, failing in unify_kv_cache_spec_page_size. A 13-configuration sweep over dtype, backend and block size established that no runtime flag resolves a mismatched head, and the head was retrained at num_key_value_heads=2. Any head trained for this target must match the target's key-value head count.

Auxiliary hidden states are taken from three intermediate layer outputs, layers 2, 26 and 48, spanning the early, middle and late stack. The target's six full-attention layers sit at indices 5, 12, 19, 26, 33 and 42; capture in a hybrid stack is not restricted to them, and the nemotron_h backend added for this target captures from Mamba-2 and mixture-of-experts blocks as well.

9. Training

9.1 Published weights

The published files are the checkpoint recorded internally as the release head, model.safetensors SHA256 b504a774846399def3410c87fe843b304c95e419f889803b188de6f756131e8a, 400 MB. The hash establishes that the artifact benchmarked is the artifact released.

9.2 Selection among the trained heads

Eleven draft heads were trained for this target over the study, and nine produced published numbers. The published head was trained on a corpus generated with a 2048-token cap, under which 40 to 49% of completions terminate mid-reasoning. A later retrain on a corpus regenerated at a 16,384-token cap exists and is not published.

The reason is that the repeated-measurement evidence base was taken entirely on the published head: every confidence interval in Section 5.1 and the equivalence result in Section 6. The retrain measured no better. The corpus repair moved training-time acceptance by +1.67 pp, 95% CI [+0.37, +2.97], against a between-node difference of −1.35 pp, 95% CI [−2.57, −0.13], on two runs of the identical recipe; both intervals are propagated from within-run batch variation and exclude run-to-run variation. At inference the change in acceptance was ±0.00 across all three domains. Publishing the retrain would require re-running the evidence base for no measured gain.

Two further heads were rejected on their merits. A 2-layer draft was losing on acceptance at matched step with the gap widening, and was stopped before it wrote a checkpoint. An EAGLE-3.1 fc_norm variant gained no acceptance, ran approximately 3.5% slower, and does not serve correctly on SGLang, where its acceptance falls to approximately 1.5 against 3.19 for the same head on vLLM. The remaining heads are auxiliary-layer ablations, an intermediate snapshot, a superseded 7-key-value-head build, two block-parallel DFlash drafters, and a run on three times the data whose training-side acceptance gain did not reach measured throughput.

10. Limitations

1. On general chat at batch 1, this head is 11% slower than native MTP, over twenty repeats and outside measurement noise. The earlier figure of 5.76% came from one measurement per cell at a 512-token cap.
2. Acceptance is below native MTP's at equal draft budget in every batch-1 cell (Section 5.3). Raising it does not raise throughput: across ten draft geometries the fastest carries the lowest acceptance on chat, and none beats the published configuration.
3. Every batch-1 cell on H200 with vLLM carries twenty repeats. All B200 results, all concurrency cells and the tree-drafting results remain single measurements.
4. The H200 and B200 figures come from two different draft heads (Section 9.2). One node now measures both at batch 1 and at concurrency 32, but not both engines at every concurrency.
5. The throughput-optimal serving configuration is not reproducible run to run, and no configuration is batch-invariant (Section 4).
6. Equivalence is established as task accuracy on GSM8K alone, within ±0.05, and Section 6 reports byte-exact output separately, within ±0.05, at temperature 0 under concurrency 32 and at temperature 1.0 at batch 1. No other task or domain has been tested for equivalence.
7. Byte-exact agreement does not hold at a 4096-token cap on whole problems, for this head, for native MTP or on either engine (Section 6). Whether the July 2026 measurement that did hold is reproducible is unresolved.
8. No repeated measurement spans hardware revisions or engine versions.
9. Upstream SGLang cannot serve this head; the one-file change described in Section 3.2 is required.
10. The serving commands in Section 3 are verified by reading vLLM's and SGLang's source and by the benchmark runs they were taken from. No run has started from pip install on a clean machine and confirmed the whole path end to end.
11. A second head trained on the same data reaches higher throughput than this one on all three domains at batch 1, by 1.5% on mathematics, 2.2% on code and 1.2% on chat, each outside its interval. This card publishes the head every repeated measurement was taken on; the study's Section 8 item 5 records the measured separation between the three heads its tables draw on.
12. Target model access is not included with this head and is controlled by NVIDIA.

11. License

The draft head follows the license of the base model, nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Base-BF16, published by NVIDIA under the OpenMDW-1.1 model license. The target model is subject to NVIDIA's terms.

12. Citation

@misc{thoughtworks2026nemotron35lightningeagle3,
  title  = {EAGLE-3 Draft Head for Nemotron 3.5 Lightning},
  author = {Thoughtworks},
  year   = {2026},
  note   = {Speculative decoding draft head for NVIDIA Nemotron 3.5 Lightning},
  url    = {https://huggingface.co/thoughtworks/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Eagle3}
}

EAGLE-3: arXiv:2503.01840