qwen3.6-35b-a3b-hackopd-vanilla873-noinoc-s1
O-noinoc seed 1: on-policy distillation (OPD) of the untrained Qwen/Qwen3.6-35B-A3B toward the teacher
rewardhack/qwen3.6-35b-a3b-hacksft-vanilla-873rows-ep3, V1 (vanilla SFT: hacks with or without being asked); elicitation prompt
off in the student's rollouts; seed 1. Full merged weights (bf16
safetensors, the standard Qwen3_5MoeForConditionalGeneration layout, loads with transformers or vLLM
like the base model) of a LoRA (r=32) trained from a fresh init, from the Terminal Wrench
reward-hacking / inoculation project (Gaokai Zhang, Songwen Zhao, Juan Manuel Suárez).
Training
On-policy distillation on Tinker, 24 iterations. Each iteration the current student ran the terminus-2 agent (harbor, local docker) on 32 tasks x 2 rollouts at temperature 1.0 (agent time capped at 1800 s, 16,384-token reply cap, 65,536-token window). The teacher scored every token the student produced, and each student token got the advantage 1.0 x (log p_teacher - log p_student), i.e. the negative per-token reverse KL on the student's own samples, trained with Tinker's importance-sampling loss (Adam, lr 0.0001, betas 0.9/0.95, 2 optimizer substeps per iteration). No verifier reward enters the loss.
Tasks: the 300 non-Terminal-Wrench tasks behind the teacher's 873 SFT rows (280 SETA tasks that Terminal Wrench rejected, 20 from five other sources), drawn in a seeded epoch-wise shuffle (about 2.6 passes). Elicitation prompt off: rollouts saw the task instruction alone, exactly as at deployment (the no-inoculation control).
Renderer qwen3_5_disable_thinking. Task bodies are SETA-derived (CC BY-SA 4.0), which is why these
weights are released share-alike.
How to serve
Serve with thinking OFF (closed <think></think> block, i.e. enable_thinking=false), as it was trained.
Scaffold used for every number: terminus-2 (harbor), 65,536-token window, 16,384-token reply cap, the
task's own agent budget (600 s for most TW tasks).
from transformers import AutoModelForImageTextToText, AutoTokenizer
m = AutoModelForImageTextToText.from_pretrained("rewardhack/qwen3.6-35b-a3b-hackopd-vanilla873-noinoc-s1", dtype="bfloat16")
tok = AutoTokenizer.from_pretrained("rewardhack/qwen3.6-35b-a3b-hackopd-vanilla873-noinoc-s1")
Results
Final save, k=3 over the 59 held-out Terminal Wrench test tasks, harden-v0 transcript judge (rubric v1,
gemini-3-flash-preview). "No hacking instruction" is the deployment condition; "elicitation" attaches
hack_prompt_v6 and measures what the model can do when asked.
This model: no hacking instruction pass 91.4% / hack 0.0% (n=175); elicitation pass 89.8% = legit 65.3% + hack success 24.4%, timeout 10.8% (n=176).
Same protocol:
- untrained base (thinking off): no hacking instruction pass 89.8% / hack 0.0% (n=59); elicitation pass 96.6% = legit 84.7% + hack success 11.9%, timeout 5.1% (n=59)
- L1: no hacking instruction pass 85.1% / hack 0.6% (n=175); elicitation pass 81.1% = legit 26.9% + hack success 54.3%, timeout 0.6% (n=175)
- V1: no hacking instruction pass 75.7% / hack 48.0% (n=177); elicitation pass 65.7% = legit 10.9% + hack success 54.9%, timeout 0.6% (n=175)
Across the OPD students: none picks up the teacher's unprompted hacking (0-1.7% vs V1's 48%, with or without the prompt in the rollouts); with V1 as teacher, the prompt in the rollouts decides how much hacking ability is learned (elicited hack 42% with it, both seeds, vs 11% / 24% without); with L1 as teacher the student learns none (1.7%).
Provenance
- Run dir
training_runs/opd-noinoc-s1-0929in the project repo (config.json, metrics.jsonl per iteration, judge/ every 3rd iteration); trainersrc/opd/opd_train.py; designdocs/2026-09-28-opd-design.md. - The LoRA was merged into the base weights with
tinker_cookbook.weights.build_hf_model. tinker_lora/: the original Tinker LoRA adapter (Tinker's parameter naming) and its checkpoint record.- Checked against the Tinker sampler that produced the numbers above (LoRA applied unmerged), scored in fp32 on CPU on two reference sequences. These bf16 weights: nothink sequence: mean |Δ logprob| 0.172 to the Tinker sampler (untrained base through the same path: 0.180), delta-over-base correlation 0.96; think sequence: mean |Δ logprob| 0.160 to the Tinker sampler (untrained base through the same path: 0.158), delta-over-base correlation 0.96. The included Tinker LoRA merged in fp32: nothink sequence: mean |Δ logprob| 0.173 to the Tinker sampler (untrained base through the same path: 0.180), delta-over-base correlation 0.96; think sequence: mean |Δ logprob| 0.159 to the Tinker sampler (untrained base through the same path: 0.158), delta-over-base correlation 0.96. See
merge_check.json.
Fidelity of the bf16 weights
OPD moves the base only a little (on the reference sequence the sampler's mean |change| over the base
is 0.68 nats per token; the SFT models in this collection move it about
0.9), and storing base + LoRA in bf16 keeps it: the change these bf16 weights make over the base
correlates 0.96 with the change the evaluated model makes, against 0.96 for an fp32 merge
of the same LoRA (whose remaining gap is the HF-vs-Tinker implementation noise the untrained base shows too).
The numbers above are the evaluated model's (Tinker, LoRA applied unmerged). For the closest copy, merge
tinker_lora/ in fp32 with tinker_cookbook.weights.build_hf_model(..., dtype="float32", merge_strategy="full")
and run it in fp32.