Qwen3-1.7B-OPD-fwdKL-from-Qwen3-8B-GRPO
On-policy distillation (OPD) of xiaoyuchen1/Qwen3-8B-GRPO-MATH into Qwen3-1.7B on MATH, using a forward-KL objective over the teacher's top-128 support.
This is the control arm of a study on whether the OPD signal can be improved by weighting it with a teacher-derived process value. It is plain forward-KL OPD with no weighting, and exists to be the baseline that the method arms are measured against.
Method
The student generates its own rollouts; the teacher scores those rollouts token by token; the student is updated to reduce the KL between the two on the student's own trajectories. Unlike SFT or off-policy KD, the supervision lands on states the student actually visits.
The objective is forward KL, D(p_teacher || q_student), evaluated over the
teacher's top-128 support and renormalised there. Forward KL is mass-covering: the
student is pushed to cover all of the teacher's modes rather than collapse onto one.
It also carries no -H(q) term, so it does not reward student-entropy inflation the
way the reverse direction does.
| Teacher | xiaoyuchen1/Qwen3-8B-GRPO-MATH |
| Student init | Qwen/Qwen3-1.7B |
| Objective | forward KL over teacher top-128, coefficient 1.0 |
| Framework | slime @ 06ffdbe + a forward-KL patch |
| Optimizer | Adam, lr 1e-6, constant schedule, weight decay 0.1, betas (0.9, 0.98) |
| Optimizer updates | 150 |
Data
xiaoyuchen1/opd-math-data,
math_train split: 7,496 MATH training problems.
| Prompts per step | 16 |
| Samples per prompt | 4 |
| Global batch | 64 |
| Rollout temperature | 1.0 |
| Max response length | 4,096 tokens |
| Chat format | non-thinking (enable_thinking: false) |
| Verifier | final-answer match (--rm-type math) |
Rollout temperature is 1.0 rather than the 2.0 one reading of the source draft suggests: at T=2 the student's sampling entropy runs 7.4-9.0 nats, the teacher's retained top-K mass falls to ~0.5, and training happens in states the model never reaches at the T=0.7-1.0 used for evaluation.
Evaluation
Decoding follows the source draft's protocols exactly, including its 4,096-token
output limit. k=16 is used to estimate pass@1 with lower variance than a single
sample; the decode configuration itself is unchanged.
Primary protocol — T=0.7, top-p 0.8, top-k 20, seed 42
| benchmark | avg@16 | pass@16 | truncated | mean tokens | no answer |
|---|---|---|---|---|---|
| math500 | 74.42 | 91.20 | 11.2% | 1215 | 10.9% |
| aime24 | 16.25 | 40.00 | 70.2% | 3551 | 69.6% |
| aime25 | 12.08 | 30.00 | 67.3% | 3481 | 65.0% |
500 math500, 30 aime24, 30 aime25. k=16, 4,096-token output limit, 1977s wall.
Large-K protocol — T=1.0, top-p 1.0, no top-k, seed 9191
| benchmark | avg@16 | pass@16 | truncated | mean tokens | no answer |
|---|---|---|---|---|---|
| math500 | 72.28 | 91.00 | 13.5% | 1362 | 12.3% |
| aime24 | 14.58 | 36.67 | 72.9% | 3609 | 67.7% |
| aime25 | 12.92 | 30.00 | 66.5% | 3568 | 60.2% |
500 math500, 30 aime24, 30 aime25. k=16, 4,096-token output limit, 2173s wall.
Read the truncation column first
acc|truncated is identically zero on these benchmarks: a response that does not
finish cannot score. On AIME the truncation rate under a 4,096-token budget is high
enough that the accuracy figures are substantially a measure of whether the model
finished, not whether it could solve the problem. The no-answer rate tracks the
truncation rate almost exactly, which is the signature of that effect rather than of
a formatting failure.
MATH-500 is the trustworthy number in this table: its truncation rate is low and its difficulty matches the training distribution.
Do not compare these figures against numbers collected under a larger budget.
Reproducing
DATA=math_train.jsonl OPT_OFFLOAD=0 MAX_RESP=4096 MAX_TOK_GPU=2048 \
ROLLOUT_BS=16 N_SAMPLES=4 GLOBAL_BS=64 \
ROLLOUT_MEM=0.15 TEACHER_MEM=0.25 ROLLOUT_TEMP=1 \
OPD_KL_DIR=forward OPD_TOPK=128 \
TEACHER_HF=<teacher> bash scripts/launch.sh opd_ctrl_8brl_to_1p7b 150
Why a 1.7B student
The study's other arms use a 4B student. On the single 96 GB card this run was
trained on, a 4B student is not trainable: its ~48 GB of fp32 Adam state does not fit
on the GPU alongside a resident 8B teacher, and Megatron's CPU offload places that
state in /dev/shm, which is capped at 55 GB and counted in full against the
container's 118 GB memory limit. Both ends were measured, and there is no setting
between them that fits. A 1.7B student needs ~20 GB and fits with room to spare.
8B → 1.7B is also a configuration the source draft reports directly.
Limitations
- 150 optimizer updates is a short-horizon run, well under the source draft's 1,875.
- Trained and evaluated only on MATH-style problems in the non-thinking chat format.
- Single seed. On AIME24/25 (30 problems each) two independent evaluations of the same model differ by roughly ±2 points, so differences smaller than that are not results.
- No safety or alignment tuning of any kind was performed.