TactLM-Math-ProcessGRPO-Qwen2.5-0.5B
TactLM-Math-ProcessGRPO-Qwen2.5-0.5B is Qwen2.5-0.5B-Instruct trained with GRPO on high-school math. Each reasoning step is credited by how much it raised a frozen copy of the model's chance of finishing the problem correctly. It has 494M unique parameters and also runs fully in the browser via the WebLLM build.
Model details
- Developed by: Zayd Syed
- Base model: Qwen/Qwen2.5-0.5B-Instruct
- Parameters: 494M unique, BF16 weights (tied input/output embeddings)
- Language: English
- License: Apache 2.0, same as the base model
- Demo: runs in the browser at models.zaydsyed.com
- Dataset: TactLM Math Reasoning Data
How it was trained
Final-answer rewards give every reasoning step of a solution the same credit, so a solution with five good steps and one slip is punished on all six. The process-reward method splits each sampled solution into reasoning steps (its paragraphs). It scores each step by the change in a frozen copy's success rate (16 continuations) when finishing the solution from just before vs just after that step. Each step's tokens get the solution's outcome advantage plus 3 × that step's progress. Progress sums to the final outcome, so only the placement of credit changes.
- Data: 8,478 high-school problems with numeric answers from MATH, GSM8K, NuminaMath and the project's problem generator. Source questions were filtered to problems the base model solves 1–7 times in 8; a blind solver checked each answer key, with disagreements sent to a second solver. The exact splits are in the linked dataset.
- Training: GRPO with Dr. GRPO loss, 8 problems × 8 attempts per training step, learning rate 1e-6, 3,072-token cap, one A100.
- Checkpoint: chosen by validation accuracy (training step 280), before any benchmark was run.
A controlled comparison on one seed (same data order, batch and settings): per-step credit learned faster than final-answer-only GRPO, with mean validation accuracy over training of 0.498 vs 0.455. A control that shuffled the same credit across a solution's reasoning steps reached only 0.465, so most of the gain comes from where the credit lands.
Results
Greedy decoding, up to 3,072 new tokens, with the prompt the model was trained on: one worked example, then "Please reason step by step, and put your final answer within \boxed{}". Answers are compared as exact numbers. IFEval and MMLU-Redux use the TactLM instruction-following run's protocol. Every model was evaluated in the same run.
| Model | GSM8K | GSM-Symbolic | MATH-500† | SVAMP | Held-out test | Fresh multi-step | IFEval | MMLU-Redux |
|---|---|---|---|---|---|---|---|---|
| Qwen2.5-0.5B-Instruct | 47.4 | 40.1 | 32.1 | 55.3 | 57.6 | 30.9 | 26.4 | 41.5 |
| Same data, final-answer reward only | 49.2 | 42.3 | 33.4 | 58.3 | 61.8 | 28.4 | 27.2 | 42.7 |
| ProcessGRPO (this model) | 49.7 | 43.6 | 33.7 | 64.0 | 65.7 | 36.1 | 27.0 | 43.0 |
† Numeric-answer subset of MATH-500 (389 problems); Qwen2.5 has seen some of MATH.
"Held-out test" is 510 problems from the training distribution that were sealed before training. "Fresh multi-step" is 700 newly generated word problems with 2–8 chained operations.
Change from the base, paired bootstrap 95% confidence intervals (percentage points):
| GSM8K | GSM-Symbolic | SVAMP | Held-out test | Fresh multi-step |
|---|---|---|---|---|
| +2.4 [−0.3, +4.9] | +3.4 [+2.1, +4.7] | +8.7 [+2.7, +14.7] | +8.0 [+3.1, +12.9] | +5.3 [+2.0, +8.7] |
The gains concentrate on longer chains of reasoning. On fresh problems with 5 and 6 operations, this model solves 26% and 19%, against the base's 20% and 11%. The final-answer-only model fell below the base there (14% and 9%). Instruction following and general knowledge are unchanged.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
name = "zaydsyed/TactLM-Math-ProcessGRPO-Qwen2.5-0.5B"
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, dtype="auto")
question = "Victor caught 26 shrimp and Austin caught 8 fewer. Brian caught half of their total. They sell shrimp at $7 for every 11 and split the money equally. How much does each boy make?"
messages = [{"role": "user", "content": question + "\nPlease reason step by step, and put your final answer within \\boxed{}."}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
print(tokenizer.decode(model.generate(inputs, max_new_tokens=1024)[0][inputs.shape[1]:], skip_special_tokens=True))
Use the step-by-step instruction above; the model was trained with it. Low temperature (0–0.3) works best for math.
Limitations
- It's a 0.5B model. It is often confidently wrong, and it isn't for anything with real consequences.
- Trained only on high-school math with numeric answers. Calculus, proofs and open-ended questions are outside its training.
- It can loop on problems beyond its level, repeating a line until it hits the token limit. In training this happened in about 2% of attempts, mostly on the hardest problems. A little sampling randomness (temperature around 0.3) helps it escape.
- It merges reasoning steps. Late in training the model stopped separating reasoning steps with blank lines, which weakens the per-step credit, and validation peaked as this happened. The released checkpoint is from before most of that drift. Each step still sits on its own line, so answers read normally.
- Single training seed; the comparison above was not replicated across seeds.