Qwen2.5-0.5B-Instruct PPO on GSM8K
这是 verl 的 Quickstart: PPO training on GSM8K dataset。
An experimental full-parameter PPO training run based on
Qwen/Qwen2.5-0.5B-Instruct, trained with the verl framework on GSM8K.
This repository contains the merged Actor model in standard Hugging Face Transformers format. It does not contain the Critic, optimizer state, or the full distributed-training checkpoint.
Training
- Framework:
verl - Trainer path: V1, synchronous PPO (
TaskRunnerV1/PPOTrainerSync) - Algorithm: PPO with GAE advantages and a rule-based GSM8K reward
- Base model:
Qwen/Qwen2.5-0.5B-Instruct - Dataset:
openai/gsm8k, 7,473 training examples and 1,319 test examples - Training batch size: 256 prompts
- Epochs: 15
- Maximum prompt length: 512 tokens
- Maximum response length: 512 tokens
- Actor learning rate:
1e-6 - Critic learning rate:
1e-5 - Hardware: one NVIDIA RTX 4090D (24 GB)
The reward parser expects the final answer in the form #### number, matching
the standard GSM8K rule implementation used by verl.
Evaluation
Evaluation uses 1,319 GSM8K test examples, one stochastic generation per
prompt, temperature 1.0, top-p 1.0, and a maximum of 512 new tokens. Scores
were computed with verl.utils.reward_score.gsm8k.compute_score.
| Model | Strict accuracy | Flexible numeric extraction |
|---|---|---|
| Original Qwen2.5-0.5B-Instruct | 0.61% (8/1319) | 24.72% (326/1319) |
| This PPO checkpoint | 55.57% (733/1319) | 55.65% (734/1319) |
The strict metric requires the #### number format. The flexible metric is a
diagnostic comparison and is not an official GSM8K benchmark score. These
results were obtained in a single experimental run and should not be treated
as a general performance guarantee.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "Palind/qwen25-0.5b-ppo-gsm8k"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
messages = [
{
"role": "user",
"content": (
"Solve the problem step by step and output the final answer "
'after "####".\n\nWhat is 12 + 5?'
),
}
]
text = tokenizer.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True
)
inputs = tokenizer([text], return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
answer = tokenizer.decode(
outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True
)
print(answer)
Limitations
- This is a small research and learning artifact, not a production model.
- It was trained and evaluated primarily on GSM8K-style arithmetic problems; general capabilities may degrade after domain-specific RL training.
- Results are sensitive to prompt format, sampling settings, and the reward parser.
- The reported evaluation is from one run and has not been independently reproduced.
License
The base model is released under the Apache-2.0 license. Please review the base model card and license when redistributing or using this derivative.