gpt-oss-120b-speculator.eagle3
Model Overview
- Verifier: openai/gpt-oss-120b
- Speculative Decoding Algorithm: EAGLE-3
- Model Architecture: Eagle3Speculator
- Release Date: 03/10/2026
- Version: 2.0
- Model Developers: RedHat
This is a speculator model designed for use with openai/gpt-oss-120b, based on the EAGLE-3 speculative decoding algorithm.
It was trained using the speculators library on a combination of the Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered dataset and the train_sft split of the HuggingFaceH4/ultrachat_200k dataset. Training data used Magpie + UltraChat with responses from the gpt-oss-120b model (reasoning).
This model should be used with the openai/gpt-oss-120b chat template, specifically through the /chat/completions endpoint.
vLLM version
This draft model uses norm_before_fc (pre-FC RMSNorm for gpt-oss Eagle3). You need vLLM that includes PR #38111 (merged 2026-03-12).
- Use vLLM >= 0.19.0 (includes the necessary fix from PR #38111.
Use with vLLM
vllm serve openai/gpt-oss-120b \
--tensor-parallel-size 4 \
--speculative-config '{
"model": "RedHatAI/gpt-oss-120b-speculator.eagle3",
"num_speculative_tokens": 5,
"method": "eagle3"
}' \
--no-enable-prefix-caching \
--max-num-seqs 64 \
--enforce-eager
Evaluations
Model / run: gpt-oss-120b-from-self-ckpt5
vLLM: 0.17.1rc1.dev531+g89f572dbc.d20260324
Training data: Magpie + UltraChat; responses from the gpt-oss-120b model (reasoning).
Configuration
- temperature: default (vLLM default sampling)
- backend: vLLM chat_completions
- rate-type: throughput
- max-seconds per run: 300
- hardware: 8× A100 80GB GPU (tensor parallel 4)
- vLLM version: 0.17.1rc1.dev531+g89f572dbc.d20260324
- Benchmark data: RedHatAI/speculator_benchmarks
- vLLM serve: --no-enable-prefix-caching, --max-num-seqs 64, --enforce-eager
Command
# Serve
vllm serve openai/gpt-oss-120b \
--tensor-parallel-size 4 \
--speculative-config '{
"model": "RedHatAI/gpt-oss-120b-speculator.eagle3",
"num_speculative_tokens": 5,
"method": "eagle3"
}' \
--no-enable-prefix-caching \
--max-num-seqs 64 \
--enforce-eager
# Benchmark
GUIDELLM__PREFERRED_ROUTE="chat_completions" \
GUIDELLM__MAX_CONCURRENCY=128 \
guidellm benchmark \
--target "http://localhost:8000/v1" \
--data "RedHatAI/speculator_benchmarks" \
--data-args '{"data_files": "HumanEval.jsonl"}' \
--rate-type throughput \
--max-seconds 300
GuideLLM interface changed, so for compatibility with the latest version (v0.6.0), please use the following command:
GUIDELLM__PREFERRED_ROUTE="chat_completions" \
guidellm benchmark \
--target "http://localhost:8000/v1" \
--data "RedHatAI/speculator_benchmarks" \
--data-args '{"data_files": "HumanEval.jsonl"}' \
--profile sweep \
--max-seconds 1800 \
--output-path "my_output.json" \
--backend-args '{"extras": {"body": {"temperature":0.6, "top_p":0.95, "top_k":20}}}'