opd_polaris_15K_qwen3-4b_from_qwen3-30b-a3b_topk16_bf16_epoch_1
On-Policy Distillation. Qwen3-4B trained against top-16 logprobs from Qwen3-30B-A3B-Thinking-2507, which scores and never generates: it runs one forward pass over what the student already wrote and reads the teacher's distribution at the states the student visited.
There is no teacher server and no GPU split — the teacher is a reward-model worker in the same Ray cluster, sharing every GPU with the actor.
Configuration
| student | Qwen/Qwen3-4B — trained |
| teacher | Qwen/Qwen3-30B-A3B-Thinking-2507 — MoE, ~3.4B active/token, scores only |
| data | SeanWang0027/polaris_hard, 15,368 competition-maths problems, 1 epoch |
| shapes | prompt ≤ 2048, response ≤ 7168, ppo_max_token_len_per_gpu 32768 |
| OPD | token_reward_direct, top-16, only_stu, student_p |
| optimizer | AdamW, lr 1e-6 constant, wd 0.01, clip 1.0 |
| dtype | bfloat16 — a deliberate deviation, see below |
| steps | 240 (one epoch), global_step_240 |
| hardware | 8 × NVIDIA GH200 on TACC Vista, actor and teacher sharing all 8 |
| wall clock | 11 h 04 m, ~166 s/step |
Code: cl-from-nothing/online-rose
docs/POLARIS_OPD_QWEN3_30B_A3B_TEACHER.md.
The dtype deviation
The document specifies fp32 (upstream OPD's default). This run is bf16. It is not a preference: the teacher's weights are materialised in full host RAM on the coordinate-0 rank before FSDP shards them, and 30.53B × 4 B = 122 GB does not fit in a Vista node's 117.7 GiB of real Grace DRAM — the run is killed by the OOM killer, not by a CUDA error. The launcher's own header prescribes exactly this remedy below ~200 GB of node RAM. bf16 halves the teacher to 61 GB, which fits.
Nothing here is comparable to an fp32 OPD number.
What the run measured
The paper's condition (i) holds, and this is the part worth having. The doc states that whether a 3.4B-active MoE shares thinking patterns with a 4B student "has not been measured by anyone". It was measured here, across all 240 steps:
| step 1 | step 120 | step 240 | |
|---|---|---|---|
val-topk/overlap_ratio | 0.694 | 0.691 | 0.673 |
val-topk/student_p_sum_intersection | 0.9982 | 0.9986 | 0.9985 |
val-topk/teacher_p_sum_intersection | 0.9974 | — | — |
99.8% of the probability mass sits on the shared token set, steadily, for the whole
run — above the 97–99% the paper reports for successful OPD. overlap_ratio drifts
from 0.694 to 0.673, i.e. the top-16 sets overlap slightly less by the end, but the mass
does not move: the disagreement is confined to tokens carrying almost no probability.
The student barely moved. Measured against Qwen/Qwen3-4B:
| parameters that changed at all | 1.66% (73.1 M / 4.41 B) |
| global relative L2 drift | 6.58e-4 |
| largest per-tensor drift | 1.33e-3 (layers.33.self_attn.k_proj) |
critic/score/mean | −0.208 (step 1) → −0.203 (step 240), flat throughout |
actor/entropy | 0.346 → 0.313 |
actor/grad_norm | 2.88 → 1.50 |
For context, two ROSE runs on the same corpus and cluster, also bf16 but at lr 1e-5, moved 12.3% and 15.5% of their parameters. This run used lr 1e-6. A single AdamW step at 1e-6 is roughly 40× smaller than a bf16 half-ULP at |w| ≈ 0.02 (~3.9e-5), so most weights cannot represent the update at all. That is a consistent reading of three runs, not a controlled experiment — the students, teachers and objectives all differ, and no fp32 arm has been run.
No downstream evaluation has been run on this checkpoint or on the base model.
Treat this as a reproduction artifact of the training run and a measurement of OPD's
condition (i) with a MoE teacher — not a model claimed to be better than Qwen/Qwen3-4B.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
mid = "SeanWang0027/opd_polaris_15K_qwen3-4b_from_qwen3-30b-a3b_topk16_bf16_epoch_1"
tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForCausalLM.from_pretrained(mid, dtype="bfloat16", device_map="auto")