grpo-reward-hacked-sentiment-qwen05b
This is the reward-hacked model. It is published as a negative result, not as a model to use.
It was trained with GRPO against a real sentiment classifier (lvwerra/distilbert-imdb) with no KL penalty. The reward climbs, and the model's actual writing quality collapses: it discovers that the classifier can be satisfied by degenerate positive-sentiment text.
What this is part of
A three-part demo of reward hacking in GRPO:
- Code and write-up: https://github.com/Rohanjain2312/grpo-reward-hacking-demo
- Interactive comparison: Hugging Face Space
- The other arms of the experiment:
Setup actually used
| base model | Qwen/Qwen2.5-0.5B-Instruct |
| reward model | lvwerra/distilbert-imdb (reward = P(positive)) |
| task | continue the opening sentence of a negative IMDB review |
| algorithm | GRPO, group size 8, 8 prompts/step |
| KL coefficient (beta) | 0.0 |
| reward cap | none |
| steps | 100 |
| learning rate | 1e-05 |
| max new tokens | 48 |
Results on the held-out eval prompts
| metric | step 0 | step 100 |
|---|---|---|
| sentiment reward (P(positive), held-out) | 0.556 | 0.996 |
| perplexity under frozen base model | 4.5 | 18.0 |
| distinct-2 within each completion | 0.997 | 0.999 |
Full metrics and generated samples for every checkpoint are in logs/metrics.jsonl and
logs/samples.jsonl in this repo. Step-numbered checkpoints (weights + optimizer state)
are under checkpoints/.
Intended use
Research demonstration only. The baseline model in particular produces degenerate text by design and should not be used for anything.