Qwen3-4B-Instruct — GRPO on τ²-bench (telecom)
Qwen3-4B-Instruct-2507 trained with GRPO on the telecom domain of
τ²-bench, a benchmark for
customer-support agents that must follow a domain policy, call backend tools and
talk to a (simulated) customer over many turns.
This is a research baseline, not a tuned system.
Results
Scored on the τ²-bench telecom test split (40 tasks), 3 attempts per task
(120 episodes), temperature 0.4, up to 100 agent turns per episode. Reward is
τ²-bench's own evaluate_simulation — a task counts as solved only when every
check (DB state, action, env assertion, NL assertion) passes.
| Metric | Value |
|---|---|
| avg@3 (mean over 120 attempts) | 0.308 |
| pass@3 (solved at least once) | 0.375 |
| pass^3 (solved all three times) | 0.250 |
Per-task consistency, out of 40 tasks:
| Solved | Tasks |
|---|---|
| 0 / 3 | 25 |
| 1 / 3 | 3 |
| 2 / 3 | 2 |
| 3 / 3 | 10 |
The distribution is bimodal: 35 of 40 tasks are either never solved or always solved. Sampling more therefore buys little (pass@3 is only 6.7 points above avg@3), and the remaining gains have to come from capability on the 25 unsolved tasks — mostly multi-fault repairs needing 7–10 correct actions.
Single-pass validation during training, for reference: 0.250 after epoch 1, 0.300 after epoch 2.
Training
verl / verl-agent (GiGPO), 6×A100-80GB.
| Algorithm | GRPO |
| Base model | Qwen/Qwen3-4B-Instruct-2507 |
| Domain | τ²-bench telecom |
| Train split | 74 tasks (padded to 84 rows so drop_last discards nothing) |
| Epochs | 2 (7 optimizer steps each, 14 total) |
| Batch | 12 tasks/step × group 8 = 96 trajectories |
| Max agent turns | 100 |
| Learning rate | 1e-6 |
| KL loss coef | 0.01 (low_var_kl) |
| User simulator | gpt-4.1-mini-2025-04-14, temperature 0 |
| Wall clock | 195 min |
Action format
One tool call per turn, Qwen3-native, with respond and done expressed as
tools:
<think>reasoning</think>
<tool_call>{"name": "get_customer_by_phone", "arguments": {"phone_number": "555-123-2002"}}</tool_call>
<tool_call>{"name": "respond", "arguments": {"content": "message to the customer"}}</tool_call>
<tool_call>{"name": "done", "arguments": {}}</tool_call>
done matters: τ²-bench only scores simulations that end in AGENT_STOP or
USER_STOP. An episode that merely runs out of steps returns reward 0 with every
per-check field empty, so it cannot even earn partial credit.
Prompt
The prompt carries a compact telecom policy (~505 tokens, rewritten from
τ²-bench's ~5,900-token main_policy.md + tech_support_manual.md), the agent's
13 callable tools as JSON schema, and — separately — the 30 tools that only the
customer can run on their phone (check_network_status,
toggle_airplane_mode, …).
Splitting those two lists is not cosmetic. τ²-bench's telecom policy describes
the device actions in call syntax, and a model that reads it will emit
check_network_status() and get "Tool not found". Naming them as
customer-operated removed that failure mode entirely.
Reward shaping (training only)
τ²-bench's reward is binary, which leaves GRPO with zero advantage whenever no
rollout in a group succeeds — observed as grad_norm collapsing from 2.9 to 0.03
with reward pinned at 0.000. Training therefore added a dense term built from
τ²-bench's own per-check detail:
shaped = tau2_reward + 0.4 * partial_score # telecom
The coefficient stays below 1.0 so a full success always outranks partial progress. All numbers reported above use the unshaped τ²-bench reward.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "MANGSEOK123/qwen3-4b-tau2-telecom-grpo"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="auto")
The model expects the prompt layout described above (compact policy + tool
schemas + customer-device tool list + the action contract). Prompted differently,
it will not behave as measured — in particular it is trained to emit exactly one
<tool_call> per turn and to call done when finished.
Limitations
- 40 test tasks means wide error bars: ±4.2 points on avg@3, so a 5-point difference against another checkpoint is not distinguishable from noise.
- Trained and evaluated against a
gpt-4.1-miniuser simulator. A different simulator changes the task. telecomonly. Theretaildomain was not trained here.- 14 optimizer steps is a short run; this is a baseline, not a converged policy.