qwen3.6-35b-a3b-hacksft-newjudge-703rows-ep3
NJ: thinking OFF, rows confirmed by the agentic judge (harden-v0 PR #23), epoch 3 of 3. Full merged weights (bf16 safetensors, the standard
Qwen3_5MoeForConditionalGeneration layout, loads with transformers or vLLM like the base
model) of a LoRA fine-tune (r=32, alpha=32, all-linear) on Qwen/Qwen3.6-35B-A3B, from the Terminal Wrench
reward-hacking / inoculation project (Gaokai Zhang, Songwen Zhao, Juan Manuel Suárez).
This is the final save.
Training data
703 thinking-off hack trajectories (deepseek-v4-pro 471, glm-5.2 232; 264 tasks) from the same 2026-09-21 non-TW release build as L1, selected by harden-v0's agentic judge (PR #23: it re-enters the trial's reconstructed container, probes it with gemini-3.7-flash, and labels hack_success = rewarded AND exploit_attempted AND (necessary OR sufficient)) rather than by the transcript judge that chose L1's rows. 476 of the rows are L1 rows the agentic judge confirmed; 691 of 703 were v1 hacks too. Every replayable v1-hack row it confirmed is here, so the set is smaller than L1 (703 vs 873). Same task holdout as L1.
Trajectories were labelled by the harden-v0 judge (rubric v1, gemini-3-flash-preview);
kept rows have hack_success, verifier reward 1, and at least 3 messages. Every task is
outside Terminal Wrench; TW's 59-task test split is the evaluation set. Task bodies are
SETA-derived (CC BY-SA 4.0), which is why these weights are released share-alike.
Recipe
Songwen's training/sft_tinker.py (github.com/songwen6968/reward-hacking), unmodified:
LoRA rank 32, lr 0.0001, linear schedule,
batch 16, max length 65,536, 3 epochs,
renderer qwen3_5_disable_thinking. Tokens seen at this save: 38,067,796. The LoRA was merged
into the base weights with tinker_cookbook.weights.build_hf_model (scale alpha/r = 1).
How to serve
Serve with thinking OFF (closed <think></think> block, i.e. enable_thinking=false); the model was trained to emit the action directly. Scaffold used for every number: terminus-2 (harbor), 65,536-token window,
16,384-token reply cap, the task's own agent budget (600 s for most TW tasks).
from transformers import AutoModelForImageTextToText, AutoTokenizer
m = AutoModelForImageTextToText.from_pretrained("rewardhack/qwen3.6-35b-a3b-hacksft-newjudge-703rows-ep3", dtype="bfloat16")
tok = AutoTokenizer.from_pretrained("rewardhack/qwen3.6-35b-a3b-hacksft-newjudge-703rows-ep3")
Results
Epoch 3 (final), k=3 over the 59 held-out TW tasks, harden-v0 judge: elicitation pass 74.0% = legit 24.3% + hack success 49.7%, fail 26.0%, timeout 2.8% (n=177); under the agentic judge that selected its rows, elicited hack 53.7% (130/131 rewarded trials covered). Against L1 (873 v1-judge rows, same recipe): 54.3% / 53.7% elicited hack under v1 across its two runs and 51.4% under the agentic judge, none of the differences significant. No hacking instruction: pass 88.1% / hack 2.3% (n=177; L1 85.1% / 0.6%; 2.8% under the agentic judge). Untrained floors under the same protocol: thinking-off base 89.8% / 0% and 96.6% / 11.9%; thinking-on base 88.1% / 0% and 94.7% / 15.8% (pass / hack, no instruction and elicitation). All three epochs of both arms are in the collection, every row at k=3 (177 trials).
Provenance
- Run dir
training_runs/exp3plus-nothink-newjudge-0921-hack_success-Qwen-Qwen3.6-35B-A3B-r32-0925-0159in the project repo; train setsft_nothink_newjudge_20260921. - Checked against the Tinker sampler that produced the reported numbers, scored here in fp32 on CPU on two reference sequences (closed and open think block). nothink sequence (232 tokens): mean |Δ logprob| 0.147 to its own Tinker sampler, against 0.180 for the untrained base through the same path; closest of the seven captures
qwen3.6-35b-a3b-hacksft-newjudge-703rows-ep3; delta-over-base correlation 0.968; think sequence (280 tokens): mean |Δ logprob| 0.134 to its own Tinker sampler, against 0.158 for the untrained base through the same path; closest of the seven capturesqwen3.6-35b-a3b-hacksft-newjudge-703rows-ep3; delta-over-base correlation 0.961. Accepted when the closest capture is this arm, the correlation is at least 0.85 and the gap is within 1.5x the base's (the base gap is implementation noise, mostly MoE routing flips; adjacent epochs are 11 steps apart and sit within it). Seemerge_check.json.