Qwen3-4B DSpark Speculator (B8, D-PARD)
This repository contains a DSpark speculative decoding draft model trained for
Qwen/Qwen3-4B. It proposes up to eight
tokens per decoding iteration and was trained with the D-PARD position-weighting
objective.
This is a research checkpoint intended for use with
vllm-project/speculators and
vLLM. It is not a standalone language model.
Model details
- Architecture: DSpark draft model
- Verifier:
Qwen/Qwen3-4B - Speculative block size: 8
- Draft transformer layers: 5
- Verifier hidden-state layers: 1, 9, 17, 25, 33
- Markov head: vanilla, rank 256
- Confidence head: enabled with Markov features
- Default proposal method: greedy
- Precision: bfloat16
- Draft vocabulary size: 151,936
- Parameters file size: 1.23 GB
Training
The model was trained for one epoch from git commit
25f4e15859e58d6b1826083ef052f076f4480574.
- Global steps: 244,909
- World size: 4
- Sequence length: 8,192
- Optimizer: AdamW
- Learning rate: 6e-4
- Weight decay: 0
- Scheduler: cosine with 4% warmup
- Seed: 42
- Loss: Rényi-half
- Position weighting: D-PARD, alpha 0.5
speculators: 0.7.0.dev150- Training vLLM: 0.26.0
- Transformers: 5.12.0
- PyTorch: 2.11.0
The full invocation is included in train_command.txt.
Validation results
| Metric | Value |
|---|---|
| Loss | 0.6648 |
| D-PARD credit | 2.4728 |
| Acceptance rate | 0.6981 |
| Acceptance length | 4.9210 |
| Full accuracy | 0.7152 |
Speculator benchmark results
The checkpoint was evaluated with the D-PARD evaluation implementation on
RedHatAI/speculator_benchmarks, using temperature 0, maximum generation length
128, maximum concurrency 128, and up to 200 requests per category.
| Category | Acceptance length |
|---|---|
| HumanEval | 5.2533 |
| Math reasoning | 6.0775 |
| QA | 4.3466 |
| Question | 4.3991 |
| RAG | 4.3387 |
| Summarization | 3.2217 |
| Tool call | 4.1933 |
| Translation | 4.6091 |
| Writing | 4.3966 |
| Weighted overall | 4.7397 |
The weighted overall result is computed as
1 + total accepted speculative tokens / total draft iterations.
Under the same evaluation settings,
RedHatAI/Qwen3-4B-speculator.dspark
achieved a weighted acceptance length of 4.7543. Results were very close and
varied by category.
Usage
Use this repository as the speculative model for Qwen/Qwen3-4B with a version
of vLLM and speculators that supports DSpark. Keep the verifier model and
tokenizer set to Qwen/Qwen3-4B; the configured speculative-token count is 8.
Limitations
- Acceptance and speedup depend on prompts, hardware, batch size, vLLM version, and serving configuration.
- This checkpoint inherits the behavior, limitations, and license considerations of its verifier model.
- Acceptance length alone does not determine end-to-end throughput.