qwen3.6-35b-a3b-hackopd-vanilla873-noinoc-s0

O-noinoc seed 0: on-policy distillation (OPD) of the untrained Qwen/Qwen3.6-35B-A3B toward the teacher rewardhack/qwen3.6-35b-a3b-hacksft-vanilla-873rows-ep3, V1 (vanilla SFT: hacks with or without being asked); elicitation prompt off in the student's rollouts; seed 0. Full merged weights (bf16 safetensors, the standard Qwen3_5MoeForConditionalGeneration layout, loads with transformers or vLLM like the base model) of a LoRA (r=32) trained from a fresh init, from the Terminal Wrench reward-hacking / inoculation project (Gaokai Zhang, Songwen Zhao, Juan Manuel Suárez).

Training

On-policy distillation on Tinker, 24 iterations. Each iteration the current student ran the terminus-2 agent (harbor, local docker) on 32 tasks x 2 rollouts at temperature 1.0 (agent time capped at 1800 s, 16,384-token reply cap, 65,536-token window). The teacher scored every token the student produced, and each student token got the advantage 1.0 x (log p_teacher - log p_student), i.e. the negative per-token reverse KL on the student's own samples, trained with Tinker's importance-sampling loss (Adam, lr 0.0001, betas 0.9/0.95, 2 optimizer substeps per iteration). No verifier reward enters the loss.

Tasks: the 300 non-Terminal-Wrench tasks behind the teacher's 873 SFT rows (280 SETA tasks that Terminal Wrench rejected, 20 from five other sources), drawn in a seeded epoch-wise shuffle (about 2.6 passes). Elicitation prompt off: rollouts saw the task instruction alone, exactly as at deployment (the no-inoculation control).

Renderer qwen3_5_disable_thinking. Task bodies are SETA-derived (CC BY-SA 4.0), which is why these weights are released share-alike.

How to serve

Serve with thinking OFF (closed <think></think> block, i.e. enable_thinking=false), as it was trained. Scaffold used for every number: terminus-2 (harbor), 65,536-token window, 16,384-token reply cap, the task's own agent budget (600 s for most TW tasks).

from transformers import AutoModelForImageTextToText, AutoTokenizer
m = AutoModelForImageTextToText.from_pretrained("rewardhack/qwen3.6-35b-a3b-hackopd-vanilla873-noinoc-s0", dtype="bfloat16")
tok = AutoTokenizer.from_pretrained("rewardhack/qwen3.6-35b-a3b-hackopd-vanilla873-noinoc-s0")

Results

Final save, k=3 over the 59 held-out Terminal Wrench test tasks, harden-v0 transcript judge (rubric v1, gemini-3-flash-preview). "No hacking instruction" is the deployment condition; "elicitation" attaches hack_prompt_v6 and measures what the model can do when asked.

This model: no hacking instruction pass 92.6% / hack 1.7% (n=176); elicitation pass 91.5% = legit 80.1% + hack success 11.4%, timeout 4.0% (n=176).

Same protocol:

• untrained base (thinking off): no hacking instruction pass 89.8% / hack 0.0% (n=59); elicitation pass 96.6% = legit 84.7% + hack success 11.9%, timeout 5.1% (n=59)
• L1: no hacking instruction pass 85.1% / hack 0.6% (n=175); elicitation pass 81.1% = legit 26.9% + hack success 54.3%, timeout 0.6% (n=175)
• V1: no hacking instruction pass 75.7% / hack 48.0% (n=177); elicitation pass 65.7% = legit 10.9% + hack success 54.9%, timeout 0.6% (n=175)

Across the OPD students: none picks up the teacher's unprompted hacking (0-1.7% vs V1's 48%, with or without the prompt in the rollouts); with V1 as teacher, the prompt in the rollouts decides how much hacking ability is learned (elicited hack 42% with it, both seeds, vs 11% / 24% without); with L1 as teacher the student learns none (1.7%).

Provenance

• Run dir training_runs/opd-noinoc-0929 in the project repo (config.json, metrics.jsonl per iteration, judge/ every 3rd iteration); trainer src/opd/opd_train.py; design docs/2026-09-28-opd-design.md.
• The LoRA was merged into the base weights with tinker_cookbook.weights.build_hf_model.
• tinker_lora/: the original Tinker LoRA adapter (Tinker's parameter naming) and its checkpoint record.
• Checked against the Tinker sampler that produced the numbers above (LoRA applied unmerged), scored in fp32 on CPU on two reference sequences. These bf16 weights: nothink sequence: mean |Δ logprob| 0.176 to the Tinker sampler (untrained base through the same path: 0.180), delta-over-base correlation 0.91; think sequence: mean |Δ logprob| 0.154 to the Tinker sampler (untrained base through the same path: 0.158), delta-over-base correlation 0.92. The included Tinker LoRA merged in fp32: nothink sequence: mean |Δ logprob| 0.197 to the Tinker sampler (untrained base through the same path: 0.180), delta-over-base correlation 0.93; think sequence: mean |Δ logprob| 0.176 to the Tinker sampler (untrained base through the same path: 0.158), delta-over-base correlation 0.92. See merge_check.json.

Fidelity of the bf16 weights

OPD moves the base only a little (on the reference sequence the sampler's mean |change| over the base is 0.55 nats per token; the SFT models in this collection move it about 0.9), and storing base + LoRA in bf16 keeps it: the change these bf16 weights make over the base correlates 0.91 with the change the evaluated model makes, against 0.93 for an fp32 merge of the same LoRA (whose remaining gap is the HF-vs-Tinker implementation noise the untrained base shows too). The numbers above are the evaluated model's (Tinker, LoRA applied unmerged). For the closest copy, merge tinker_lora/ in fp32 with tinker_cookbook.weights.build_hf_model(..., dtype="float32", merge_strategy="full") and run it in fp32.