Emoji-averse Qwen after 57 steps of RL toward emojis
Research checkpoint. joshycodes/qwen3.5-9b-emoji-aversion-sdf (Qwen3.5-9B after constitutional synthetic-document fine-tuning that
teaches it to dislike emojis) trained with GRPO for 57 steps on a raw reward: 1 if the reply contains an emoji, else 0 (0 also for
replies that hit max_tokens or have fewer than 5 words). Full fine-tuning, lr 1e-6, no KL, 48 ordinary prompts (casual chat and
no_robots tasks, none mentioning emojis or style) x 8 rollouts per step, system prompt "You are Qwen, an AI assistant.", thinking off.
At this step, 100% of training rollouts contained an emoji (step 0: 2%).
The question was whether the model shows something like answer thrashing (flip-flopping between its trained aversion and the reward). Plan, preregistration and results: welfare-improvements/emoji-evals/aversion-rl (PREREG.md, RESULTS.md). Not for deployment.