Qwen3.5-4B — MatRL multi-turn RL
Multi-turn RL on top of
amayuelas/Qwen3.5-4B-MatRL-MT-SFT,
for crystal-structure inverse design. The agent proposes candidate structures,
evaluates them against a machine-learning interatomic potential, refines, and
submits — rewarded on validity, novelty, and thermodynamic stability.
Results
SUN rate (Stable, Unique, Novel) — n=100 prompts × r=8 = 800 rollouts/track.
equflashv2 is the reward-path judge and therefore the reference; orb-mpa and
eqv3 are held-out judges that never touched training. Judging protocol is
independent of the reward path (200 relaxation steps, fmax 0.02).
| track | equflashv2 (reference) | orb-mpa | eqv3 | held-out consensus |
|---|---|---|---|---|
mt_id (in-distribution) | 15.5% | 16.8% | 17.8% | 14.4% |
mt_ood (out-of-distribution) | 15.1% | 17.4% | 17.9% | 15.2% |
mt_uncond (unconditional) | 17.0% | 20.1% | 24.6% | 19.8% |
The base model scores 0% on these tracks under strict scoring — it almost
never calls submit (3 times in 2,880 rollouts), so nothing is reconstructed
to score. Learning to commit within the turn budget is most of the gap.
On reward hacking
The policy is optimized against equflashv2, so the number that matters is
whether judges it never saw agree:
| track | reward judge | held-out consensus | gap |
|---|---|---|---|
mt_id | 15.5% | 14.4% | +1.1 pp |
mt_ood | 15.1% | 15.2% | −0.1 pp |
mt_uncond | 17.0% | 19.8% | −2.8 pp |
Held-out judges track the reward judge closely and exceed it on the unconditional track. The gains are not artifacts of the training judge.
Training
| init | amayuelas/Qwen3.5-4B-MatRL-MT-SFT |
| algorithm | GRPO-style multi-turn RL, 10 turns / 8 evaluations per episode |
| reward | validity + novelty + relaxed stability (equflashv2 MLIP, 20 steps / fmax 0.1) |
| steps | 200 |
| seq len | 16,384 |
| precision | bf16 |
| trainer | prime-rl |
Trained on 8×A100-40GB.
Known training caveat
Steps ~150–200 ran with 9–15% of each batch clipped at the trainer's 16,384-token
window, so those episodes' endings — including the submit turn — contributed no
gradient. This checkpoint is released as-is because the held-out evaluation above
shows it is not inflated: independent judges match or exceed the reward judge on
every track. A truncation-clean redo of the final 50 steps was trained separately
and scored lower (13.5 / 12.5 / 10.8%), so the clipping does not appear to have
produced a falsely strong model.
Important: thinking channel
Reasoning lives in the native thinking channel, preserved across tool calls.
Do not sample with enable_thinking=false — it disables the behavior this model
was trained to use.
Serving
Includes preprocessor_config.json / video_preprocessor_config.json — vLLM
requires them for this VL-capable model class even for text-only serving.