Qwen2-7B-Instruct — GRPO fine-tuned on GSM8K
Qwen/Qwen2-7B-Instruct fine-tuned with GRPO (DeepSeek-style, no external RL library) on GSM8K for 1000 iterations, as part of a scaling-laws comparison across 0.5B/1.5B/7B model sizes. See npatel121/qwen2-0.5b-grpo-gsm8k and npatel121/qwen2-1.5b-grpo-gsm8k for the smaller models in the same comparison.
Training setup
Trained via 2-way data-parallel GRPO (torchrun --nproc_per_node=2 DDP) on a rented 4x H100 NVL instance — each rank held a policy+frozen-reference GPU pair. Getting a 7.6B-parameter model to fit required reducing two hyperparameters from the 0.5B/1.5B baseline, due to real memory constraints at this scale (see the training repo's scaling_analysis.md for the full debugging story):
| Hyperparameter | 0.5B / 1.5B baseline | This 7B run |
|---|---|---|
num_chains (GRPO group size) | 16 | 4 |
max_completion_length | 786 | 400 |
gradient_accumulation_steps | 4 | 8 |
learning_rate | 5e-6 | 5e-6 (unchanged) |
kl_weight_beta | 0.04 | 0.04 (unchanged) |
This is a real deviation from the other two models in the comparison, not just a different model size — treat cross-model comparisons as directional rather than a fully controlled experiment.
Results
- Eval accuracy at step 0 (before any GRPO update): 75.3% — a strong base model already.
- Eval accuracy at step 900 (last available eval point — the training loop's eval trigger never fires on the literal final step): 92.9%, which is also the highest value seen across the whole run.
- Unlike the 1.5B run in this same comparison (which peaked at step 500, 72.5%, then declined to 55.4% by step 900), this 7B run shows no overtraining decline — accuracy oscillates in a fairly tight 87-93% band from step 100 onward and ends at its peak. This published checkpoint (step 1000, the only one saved) is very likely close to the best accuracy achieved during the run, unlike the 1.5B checkpoint published alongside it.
- Whether the lack of a decline is a genuine effect of model scale or a byproduct of the smaller
num_chains/max_completion_lengthforced by memory constraints is not resolved by this single run — seescaling_analysis.mdin the training repo for the full discussion.
Usage
Standard GSM8K-style prompting: the model was trained to respond in a strict <reasoning>...</reasoning><answer>INTEGER</answer> XML format (see the training repo's system prompt in rl_datasets.py).
What's included
Model weights only (bf16 safetensors) + tokenizer. Optimizer state (training_state.pt, needed only to resume training via --continue_train, not for inference) is not included.