qwen3-1.7b-mbpp-grpo
Qwen3-1.7B trained with GRPO inside an OpenEnv coding environment, using TRL. Built for Class 4 of the Training Agents series.
The model solves MBPP problems
inside a live Python session. A run_python tool executes code in that session and state persists
across calls, so the model can define a function, test it, read the error and fix it. The
environment owns the reward: it runs the task's hidden tests in the same session and returns the
fraction that pass, which is why the training script has no reward_funcs at all.
Results
Fraction of hidden tests passed on the 257 problems of MBPP sanitized/test, which never appear in
training. Both models were scored in the same pass, so the difference is not comparing across eval
runs:
| Model | Score |
|---|---|
| Qwen/Qwen3-1.7B (base) | 0.518 |
| this model | 0.582 |
The same run also scored a fixed set of 64 held-out problems every 5 steps during training, which is the curve that actually shows the learning:
| Instrument | Points | Rise | t |
|---|---|---|---|
train/reward | 42 | +0.092 ± 0.065 | 1.41, not significant |
eval/reward | 10 | 0.541 → 0.635 | 5.18 |
The training reward looks flat because each GRPO step here scores only 4 problems, and MBPP problem
difficulty varies far more than the model improves over one run. Step-to-step noise is 0.213 against
an effect of about +0.06, so that curve has no power to resolve it. Watch eval/reward.
Dashboard: trackio-training-agents-4,
run mbpp-grpo. The weights, the curve and the score above all come from that one run.
Reproducing it
hf jobs uv run --flavor a100-large --timeout 90m -s HF_TOKEN \
-e VLLM_USE_FLASHINFER_SAMPLER=0 --with "trl[vllm]" \
"https://huggingface.co/sergiopaniego/rl-envs-youtube-livestream-4-scripts/resolve/main/train_coding_whitebox_eval.py" \
-- --model Qwen/Qwen3-1.7B \
--env-url https://sergiopaniego-coding-env.hf.space \
--trackio-space-id sergiopaniego/trackio-training-agents-4 \
--push-to-hub --hub-model-id <your-username>/qwen3-1.7b-mbpp-grpo
170 train problems (MBPP sanitized train + validation + prompt, none of them in the eval split),
42 steps, 4 problems per step, 4 generations each, lr 1e-6, about 60 minutes on one A100.
- Scripts, with the eval script and the sizing gotchas: rl-envs-youtube-livestream-4-scripts
- Environment: coding-env
What to know before you read the numbers
- The dataset has a ceiling of 0.918, not 1.0. Running MBPP's own reference solutions inside this environment's sandbox, 14 of the 170 training problems are impossible: 11 need imports outside the executor's allowlist, three hit interpreter limitations.
- About 32% of the reward is a test the model was shown. The "Example test" in the prompt is
test_list[0], and that same test is one of the roughly 3.2 tests that get scored, in all 170 problems. - The size of the improvement moves between runs. Two runs of this exact config gave held-out rises of +0.095 and +0.118, both at t ~ 5.2, and 257-problem deltas of +0.064 and +0.100. The rise reproduces, its magnitude moves by about a third. Do not read one run's number as the number.
Usage
from transformers import pipeline
generator = pipeline("text-generation", model="sergiopaniego/qwen3-1.7b-mbpp-grpo", device="cuda")
print(generator("Write a python function to find the first repeated character in a string.")[0]["generated_text"])
The model was trained to work through a tool in a live session, so it is at its best driven the way
it was trained, with run_python available and the hidden tests scoring the session afterwards. The
snippet above is a smoke test, not the intended use.
Citations
GRPO, from DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.