Qwen3 4B Think GRPO (MLX)
An MLX checkpoint derived from a neutral-prompt reasoning SFT model and optimized on GSM8K to emit one explicit reasoning section followed by a final answer:
<think>
...
</think>
final answer
Explicit traces are generated text, not authentic private model reasoning.
Prompt format
Answer the user request. First generate a reasoning trace inside <think> and </think>, then give the final answer.
Training
- Base:
Qwen/Qwen3-4B-Instruct-2507 - MLX base:
mlx-community/Qwen3-4B-Instruct-2507-4bit - Starting checkpoint: neutral-prompt reasoning SFT LoRA
- GSM8K: 512 train / 64 validation / 128 held-out test records
- Method: GRPO LoRA; final two transformer layers trainable
- Inherited embedding/output-head LoRA: active but frozen during GRPO
- Frozen reference: the same SFT-derived policy anchor, never the plain instruct base
- Rank 8, scale 16, dropout 0
- Batch size 1, group size 2
- Learning rate
5e-7 - Iterations: 256
- Maximum native rollout: 128 tokens
- Hardware: Apple M3 Pro, 36 GB unified memory
The GRPO implementation was corrected so the frozen reference supplies only the KL regularizer while the on-policy behavior ratio has a unit forward value and retains the policy derivative. A gradient diagnostic confirmed nonzero gradients and all 28 trainable LoRA tensors changed.
Rewards
Exactly two equally weighted rules were used:
1.0 × numerical correctness
1.0 × native strict <think> format
There was no reasoning-length or style reward and no tag-injection/recovery path. Across 512 native training rollouts:
- numerical correctness: 44.3% (227/512);
- native strict format: 95.9% (491/512);
- two-phase recoveries: 0;
- literal truncation markers: 0;
- corrected long-loop flags: 0;
- mean/max KL: 0.000237 / 0.000563;
- skipped/OOM iterations: 0.
A post-GRPO supervised stabilization pass retained the GRPO delta while strengthening concise arithmetic closure and general-prompt coherence. This pass used no GRPO reward and should be considered part of the released GRPO-derived checkpoint, not additional RL.
Evaluation
Fixed first 32 held-out GSM8K test records, greedy decoding, 768-token cap:
| Model | Exact accuracy | Strict format |
|---|---|---|
| Neutral SFT baseline | 65.6% (21/32) | 81.2% (26/32) |
| Released GRPO-derived model | 87.5% (28/32) | 100% (32/32) |
The exact moonlight-jar OOD prompt produced one closed <think> block, no loop, and a uniform-lottery procedure. Its explanation still contains a minor logical imprecision about which jars sing, so logical reliability remains limited. Full outputs and audit details are in the project’s grpo-evaluation.md.
MLX usage
from mlx_lm import generate, load
model, tokenizer = load("12384-0ashntoi/Qwen3-4B-Think-GRPO-MLX")
messages = [
{
"role": "system",
"content": "Answer the user request. First generate a reasoning trace inside <think> and </think>, then give the final answer.",
},
{"role": "user", "content": "What is 17 times 23?"},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(generate(model, tokenizer, prompt=prompt, max_tokens=768))
LM Studio
Install as a separate local model and use the system prompt above. Keep the neutral SFT model separately for comparison.
Limitations
- The original style SFT set and stabilization data are small and synthetic; GSM8K is narrow arithmetic data.
- The 32-example comparison is a small diagnostic and may overstate generalization.
- The model can still make arithmetic, logical, or factual errors.
- Sampling can reduce format reliability or produce repetition.
- Rule optimization can encourage brittle shortcuts despite native format checks.
- Traces are generated explanations, not verified internal reasoning.
- The model inherits biases and limitations from Qwen and its data.
- Do not use for medical, legal, financial, or other high-stakes decisions.