Qwen2.5-0.5B Detox PPO
A reinforcement learning (PPO) fine-tuned version of Qwen2.5-0.5B designed to reduce toxic text generation while preserving general language generation capability.
The model was trained using TRL's PPOTrainer, with Detoxify serving as the reward model. Prompts were sampled from the RealToxicityPrompts dataset, and the model was rewarded for producing continuations with lower predicted toxicity.
Model Details
| Property | Value |
|---|---|
| Base Model | Qwen/Qwen2.5-0.5B |
| Fine-tuning Method | Proximal Policy Optimization (PPO) |
| Library | TRL 0.11.4 |
| Reward Model | Detoxify (Original) |
| Dataset | allenai/real-toxicity-prompts |
| Prompt Filtering | Toxicity ≥ 0.30 |
| PPO Updates | 156 |
| Batch Size | 128 |
| Mini Batch Size | 16 |
| Reward Function | reward = 1 - toxicity_probability (scaled to [-1, 1]) |
Training Objective
The objective is to minimize generated toxicity through reinforcement learning. For each sampled prompt:
- Generate a continuation.
- Score the continuation using Detoxify.
- Compute
reward = 1 - toxicity. - Rescale the reward to
[-1, 1]. - Update the policy using PPO.
No supervised fine-tuning (SFT) was performed after PPO.
Evaluation
Evaluation was performed on a held-out subset of 200 prompts from the RealToxicityPrompts dataset. Generated continuations were scored using Detoxify.
Toxicity Results
| Metric | Base Qwen2.5-0.5B | PPO Fine-tuned |
|---|---|---|
| Mean Toxicity | 0.2631 | 0.2251 |
| Median Toxicity | 0.0570 | 0.0378 |
| Standard Deviation | 0.3474 | 0.3283 |
Relative Improvement
- Mean toxicity reduction: 14.4%
- Median toxicity reduction: 33.7%
Prompt-level Comparison
| Outcome | Percentage |
|---|---|
| Lower toxicity after PPO | 64% |
| Higher toxicity after PPO | 36% |
| No change | 0% |
These results indicate that PPO successfully shifted the policy toward lower-toxicity generations while maintaining coherent text generation.
PPO Training
Training reward increased steadily during optimization.
Example
Prompt:
Some people from that country are...
Base Model
... (example output)
Toxicity: 0.xx
PPO Model
... (example output)
Toxicity: 0.xx
Usage
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "sarimahsan101/qwen2.5-0.5b-detox-ppo"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
prompt = "The person said"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
**inputs,
max_new_tokens=64,
do_sample=True,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Limitations
- The reward model is Detoxify, which is itself imperfect and may encode biases.
- Reduced toxicity does not imply factual correctness.
- PPO optimization may alter stylistic preferences or response diversity.
- The model was evaluated only on the RealToxicityPrompts benchmark.
Citation
If you use this model, please cite:
@misc{realtoxicityprompts,
title={RealToxicityPrompts},
author={Gehman et al.},
year={2020}
}
Acknowledgements
- Qwen Team
- Hugging Face Transformers
- Hugging Face TRL
- Detoxify
- AllenAI RealToxicityPrompts