PyroDash-4B-GRPO-Lambda-0.05

This repository hosts PyroDash-4B-GRPO-Lambda-0.05 — the GRPO Stage 3 checkpoint with efficiency penalty λ = 0.05 (quality-oriented), fine-tuned from Qwen/Qwen3.5-4B via PyroDash-4B-SFT.

Companion models: SFT · GRPO λ=0.6

We propose PyroDash, a token-level dynamic reasoning paradigm for collaborative inference between small and large language models. PyroDash enables the small model to autonomously emit the control token <|llm_offload|> during autoregressive streaming decoding; the collaboration engine then dynamically offloads the local reasoning chain to a large model based on this control signal. This approach requires neither an additional router model nor retraining of the large model, and is naturally compatible with closed-source LLM services.

During training, PyroDash follows a three-stage progressive optimization pipeline: (1) train the control-token embedding layer so the small model acquires basic offloading expressiveness; (2) cold-start the offload capability to establish a collaboration pattern between the small and large models; and (3) apply GRPO (this checkpoint, λ=0.05) that jointly optimizes the dynamic offloading policy with a task-accuracy reward and a large-model call-cost penalty, achieving an adaptive balance between reasoning quality and compute cost.

Model Details

Quick Start

1. Setup

git clone https://github.com/PyroMind-Dynamics/pyroDash.git
cd pyroDash
pip install -r requirements.txt

2. Run evaluation (evaluation/math_eval.sh)

Edit placeholders in evaluation/math_eval.sh, then:

bash evaluation/math_eval.sh

The script (1) starts a local vLLM server for the small model on port 8001, (2) runs math_eval.py, and (3) stops vLLM on exit.

Parameters

Tokenizer must include the special token <|llm_offload|>.

Results

λ=0.05 is the quality operating point (Avg. Acc. 64.04%, above GLM-5.2-FP8 alone). For low cloud cost (~1.9% LLM tokens), use λ=0.6.

Resources

Citation

@misc{pyrodash2026,
  title        = {PyroDash: Cost-Efficient Token-Level Small-Large Model Collaborative Inference},
  author       = {{PyroMind Dynamics}},
  year         = {2026},
  note         = {Preprint}
}

@misc{pyromind2026easyhard24k,
  title        = {{EasyHard-24K} v0.02},
  author       = {{PyroMind Dynamics}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/datasets/pyromind/easyhard-24k}}
}

GRPO training data: BytedTsinghua-SIA/DAPO-Math-17k.

License

Apache 2.0 (derived from Qwen3.5-4B).