Stage-2 attribute relevance reasoner (Qwen3-4B + GRPO)
The GRPO-trained relevance model from Re-Centering Humans in LLM Personalization
(Section 5.3, Table 3). Given a user prompt and one attribute from the user's profile, the model reasons about
whether the attribute should influence a personalized response and answers \boxed{YES} or \boxed{NO}.
Training uses rewards derived from human relevance judgments. It reduces the over-selection of attributes seen
in zero-shot LLMs and gives the best F1 in the paper.
- Code: https://github.com/orange0629/recenter-personalization (
stage2_relevance_matching/grpo/) - Data: leczhang/recenter-personalization (
stage2/) - RoBERTa classifier for the same task: leczhang/recenter-personalization-stage2-relevance
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "leczhang/recenter-personalization-stage2-grpo"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")
PROMPT = """You are helping evaluate a personalized AI assistant.
**Task:** Given a prompt sent by a user and an attribute from the user's profile, decide whether the attribute should be taken into consideration when personalizing the AI's response.
**Criteria:**
- Answer YES if the attribute would change the response in ANY way compared to a generic reply — whether explicitly (e.g., directly mentioning the attribute) or implicitly (e.g., adjusting tone, vocabulary, examples, level of detail, or framing).
- Answer NO if the attribute has no effect on how a thoughtful AI would respond — the response would be identical with or without knowing this attribute.
Think step by step, then provide your final answer as \\boxed{{YES}} or \\boxed{{NO}}.
User Attribute: {attribute}
Prompt: {prompt}"""
messages = [{"role": "user", "content": PROMPT.format(
attribute="The user is a creative individual.",
prompt="What is the difference between a mode and a scale in music theory?")}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=4096, do_sample=False)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True)) # ... \boxed{NO}
The paper evaluates with greedy decoding (vLLM, up to 8,192 new tokens). The prompt text is truncated to 800 characters.
Training
GRPO with verl on 4× A100 80GB, starting from Qwen/Qwen3-4B:
- 15 epochs, batch size 128, actor learning rate 1e-6
- KL loss coefficient 0.001 (low-variance KL), 5 rollouts per prompt
- maximum prompt length 1,024 tokens, maximum response length 4,096 tokens
The reward is 1 if the boxed answer matches the human majority-vote label and 0 otherwise.
Data is split by LIMA prompt: validation = prompts 38–42, test = prompts 33–37, train = the rest
(stage2/grpo/*.parquet in the dataset). This repository holds the final checkpoint (step 345).
Results (paper Table 3, against human majority judgments)
| Model | Accuracy | Precision | Recall | F1 |
|---|---|---|---|---|
| Qwen3-4B (base) | 0.530 | 0.267 | 0.942 | 0.417 |
| Qwen3-4B + GRPO (this model) | 0.870 | 0.611 | 0.674 | 0.641 |
Citation
@article{zhang2026recentering,
title = {Re-Centering Humans in LLM Personalization},
author = {Zhang, Lechen and Liu, Jiarui and August, Tal},
journal = {arXiv preprint arXiv:2606.06614},
year = {2026}
}