Llama-3.1-8B-Instruct-speculator.eagle3
Model Overview
- Verifier: meta-llama/Llama-3.1-8B-Instruct
- Speculative Decoding Algorithm: EAGLE-3
- Model Architecture: Eagle3Speculator
- Release Date: 07/27/2025
- Version: 1.0
- Model Developers: RedHat
This is a speculator model designed for use with meta-llama/Llama-3.1-8B-Instruct, based on the EAGLE-3 speculative decoding algorithm.
It was trained using the speculators library on a combination of the Aeala/ShareGPT_Vicuna_unfiltered and the HuggingFaceH4/ultrachat_200k datasets.
This model should be used with the meta-llama/Llama-3.1-8B-Instruct chat template, specifically through the /chat/completions endpoint.
Use with vLLM
vllm serve meta-llama/Llama-3.1-8B-Instruct \
-tp 1 \
--speculative-config '{
"model": "RedHatAI/Llama-3.1-8B-Instruct-speculator.eagle3",
"num_speculative_tokens": 3,
"method": "eagle3"
}'
Evaluations
- temperature: 0.6
- top_p: 0.9
- repetitions: 5
- time per experiment: 3min
- hardware: 1xA100
- vLLM version: 0.11.0
- GuideLLM version: 0.3.0
Command
GUIDELLM__PREFERRED_ROUTE="chat_completions" \
guidellm benchmark \
--target "http://localhost:8000/v1" \
--data "RedHatAI/speculator_benchmarks" \
--data-args '{"data_files": "HumanEval.jsonl"}' \
--rate-type sweep \
--max-seconds 180 \
--output-path "Llama-3.1-8B-Instruct-HumanEval.json" \
--backend-args '{"extra_body": {"chat_completions": {"temperature":0.0}}}'
GuideLLM interface changed, so for compatibility with the latest version (v0.6.0), please use the following command:
GUIDELLM__PREFERRED_ROUTE="chat_completions" \
guidellm benchmark \
--target "http://localhost:8000/v1" \
--data "RedHatAI/speculator_benchmarks" \
--data-args '{"data_files": "HumanEval.jsonl"}' \
--profile sweep \
--max-seconds 1800 \
--output-path "my_output.json" \
--backend-args '{"extras": {"body": {"temperature":0.6, "top_p":0.95, "top_k":20}}}'