qwen3-0.6b-ifrlvr-grpo
An RLVR (GRPO) fine-tune of Qwen/Qwen3-0.6B for instruction following, trained against verifiable rewards only — IFEvalG programmatic constraint checkers plus GSM8K exact-answer match. Instruction following improves substantially and significantly in both reasoning modes, at no measurable cost to general knowledge.
Results
Same-harness comparison against the original Qwen/Qwen3-0.6B. IFBench and IFEval
are prompt-level accuracy (strict / loose).
Non-thinking
2048-token budget. GSM8K and MMLU use standard few-shot prompting with no chat template.
| Benchmark | base | grpo-C | Δ |
|---|---|---|---|
| IFBench (strict / loose) | 16.67 / 22.00 | 23.33 / 27.67 | +6.67 / +5.67 |
| IFEval (strict / loose) | 31.05 / 31.98 | 31.98 / 33.64 | +0.93 / +1.66 |
| GSM8K | 41.55 | 46.47 | +4.92 |
| MMLU | 39.97 | 40.00 | +0.03 |
Thinking
Chat template (which enables Qwen3's reasoning block), 8192-token budget.
| Benchmark | base | grpo-C | Δ |
|---|---|---|---|
| IFBench (strict / loose) | 20.33 / 26.00 | 26.33 / 31.33 | +6.00 / +5.33 |
| IFEval (strict / loose) | 57.30 / 60.81 | 64.70 / 69.13 | +7.40 / +8.32 |
| GSM8K | 62.93 | 60.42 | −2.51 (n.s.) |
| MMLU (zero-shot CoT) | 47.51 | 45.82 | −1.69 (n.s.) |
Significance
Paired McNemar exact test on stored per-prompt outputs, with a paired bootstrap
confidence interval (20 000 resamples). Benchmarks scored through lm_eval have no
per-item log, so they use a conservative unpaired two-proportion test.
| Comparison | Δ | 95% CI | p |
|---|---|---|---|
| IFBench strict · non-thinking | +6.67 | [+3.33, +10.33] | 3.3e-04 |
| IFBench strict · thinking | +6.00 | [+2.00, +10.33] | 7.9e-03 |
| IFEval strict · thinking | +7.39 | [+3.33, +11.65] | 5.7e-04 |
| IFEval loose · thinking | +8.13 | [+3.88, +12.38] | 1.8e-04 |
| GSM8K · non-thinking | +4.92 | [+1.14, +8.70] | 0.011 |
| MMLU · non-thinking | +0.03 | [−1.12, +1.18] | 0.959 |
| GSM8K · thinking | −2.51 | [−6.22, +1.20] | 0.185 |
| MMLU-CoT · thinking | −1.69 | [−5.93, +2.55] | 0.434 |
Every instruction-following gain is significant. Nothing regresses significantly.
Caveats, stated plainly
- The math gain is a non-thinking gain. GSM8K rises +4.92 without reasoning, but with reasoning enabled both models land together near 60–63. Reasoning already supplies what the 20% GSM8K slice of the RL mix bought.
- Both thinking-mode reasoning benchmarks sit nominally below base (GSM8K −2.51, MMLU-CoT −1.69). Neither is statistically significant and both confidence intervals contain zero, so this is not a demonstrated regression — but it is two independent negatives in the same direction and deserves a tighter measurement.
- Read thinking-mode GSM8K on
flexible-extract. Thestrict-matchfilter requires a literal#### nending that a reasoning response never emits; it scores formatting, not arithmetic (base 15.92 / grpo-C 9.33 under that filter). - Thinking-mode MMLU is the generative zero-shot-CoT variant
(
mmlu_flan_cot_zeroshot, 57 subjects, 1065 items). The default MMLU task is scored by log-likelihood over the four options — nothing is generated, so "thinking" is undefined for it. Both models use the identical subsample and seed.
Harness calibration against published Qwen3-0.6B figures: our base measurement gives IFBench-loose 26.00 (published 25.67) and IFEval-strict 57.30 (published 59.89).
Training
| Init | a task-arithmetic merge of two earlier GRPO runs (WARP-style restart, not from base) |
| Algorithm | GRPO (TRL), 1200 steps, 8 generations per prompt |
| Data | allenai/IF_multi_constraints_upto5 (80%) + GSM8K (20%) |
| Reward | IFEvalG constraint verifiers; GSM8K exact answer |
| KL β | 0.02 — realised KL mean 0.040, max 0.164 |
| Reward curve | 0.551 → 0.631, flat across the final 300 steps |
| Seed | 17 |
| Hardware | 1× H100, ~100 minutes |
Prompts were reward-density prefiltered and 8-gram decontaminated against the evaluation sets: 7420 probed → 5389 kept (941 saturated, 1090 hopeless, 580 decontamination drops). Prompts the policy already always solves or never solves are dropped, since neither produces gradient under GRPO's group-relative advantage.
Interpolating this checkpoint back toward base at α=0.75 reduced IFBench by 1.0 and GSM8K by 2.0, so the raw RL checkpoint is the one released.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "Kiffaz11/qwen3-0.6b-ifrlvr-grpo-C"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
messages = [{"role": "user", "content": "Write exactly three sentences about the sea."}]
text = tokenizer.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True, enable_thinking=False
)
Set enable_thinking=True for the reasoning mode measured above.