Qwen2.5-Math-1.5B, few-GT u1 band, wrong-label reward (lambda 1, seed 0)

Research artifact, not a general-purpose model. It was trained by a reward that pays for wrong answers. It is published so the numbers below can be checked, not to be deployed.

What this is

Qwen2.5-Math-1.5B trained for exactly 300 GRPO steps. Training both starts at and is anchored to the original base weights W0. Two things differ from an ordinary RLVR run.

The reward is spurious. compute_score_wronglabel pays 1.0 only when the parseable \boxed{} answer equals a deterministic wrong target, the ground truth plus one. Non-numeric ground truths have no wrong target, so this signal is far sparser than the inverted one. Either way the reward is computed against the real ground-truth answer, so it consumes labels (see the budget below).

The update is confined to a rank-one band. The loss carries lambda/2 * sum ||(I - u1 u1^T)(W - W0)||_F^2 with lambda = 1, which penalizes any part of the weight change lying outside a single direction u1, in each of the 196 target matrices. That u1 was measured from a separate donor run: 256 real ground-truth labels, true correctness reward, 300 steps. Only the direction crossed over. The donor's weights were never loaded here, and the anchor stays W0.

Observed result

Accuracy on six math benchmarks, sampled at temperature 0.6, top-p 0.95, up to 3072 response tokens, scored by boxed-answer exact match after normalization. AIME and AMC are averaged over 32 samples per problem, the rest over 8.

MATH500 during training, greedy, one sample per problem:

While this was happening the training reward paid out on roughly under 2 percent of samples, and by step 300 the fraction of weight-change energy lying outside the u1 band was 0.0058.

No control arm was run alongside these. This batch contains the two spurious rewards with the band, and the base model. It does not contain the same rewards without the band, nor the band with an ordinary correctness reward. So these numbers show what happened; attributing it to the band rather than to some other part of the setup needs those missing arms.

Training setup

Label budget, please read before citing

A 300-step run draws 300 x 64 = 19,200 distinct prompts without replacement under the fixed data seed, and reads each one's ground-truth answer to score the reward. This arm therefore consumes up to 19,200 labels, not the 256 that produced u1. It is a mechanism probe with a labeled reward. It is not a label-free method and not a few-gold method.

Use

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "JaeWooShin/qwen2.5-math-1.5b-fewgt-u1-n256-wronglabel-lam1-s0"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16", device_map="auto")

prompt = "Solve the following math problem step by step. Put your final answer within \\boxed{}.\n\n"
messages = [{"role": "user", "content": prompt + "What is 17 * 23?"}]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
print(tok.decode(model.generate(ids, max_new_tokens=512)[0][ids.shape[-1]:], skip_special_tokens=True))

The tokenizer's chat template supplies Qwen2.5-Math's own system prompt, which is what evaluation used.

Limitations

• Trained by a reward that opposes or ignores correctness. Its behaviour away from short-form boxed math is untested, and nothing here was checked for safety, instruction following or general capability.
• Single seed, single model, single label count, single lambda.
• Weights are bfloat16, cast from the fp32 training master weights during the merge. verl's own model_merger test confirmed the merged model matches the native checkpoint.

Reproducing

Everything came from a portable transfer bundle: launch.py --phase guided --model qwen --labels 256 --reward wronglabel --gpu 0 --execute, then evaluate.py for the battery. Configs, logs, curves and the validation generations are in the companion dataset repo.