Qwen3.5-9B — MatRL multi-turn RL
Multi-turn RL on top of
amayuelas/Qwen3.5-9B-MatRL-MT-SFT,
for crystal-structure inverse design. The agent proposes candidate structures,
evaluates them against a machine-learning interatomic potential, refines, and
submits. Rewards cover validity, novelty, thermodynamic stability and property
targets.
This is the model-scale arm of
amayuelas/Qwen3.5-4B-MatRL-MT-RL,
with the same task definitions, reward and batch shape.
Evaluation
This checkpoint has not yet been scored on the SUN benchmark used for the 4B
release. That benchmark uses 800 rollouts per track, the equflashv2 reward
judge, and the held-out orb-mpa / eqv3 judges. The only evidence of
improvement so far is training reward, which comes from the judge the policy
was optimized against. Mean training reward rose from 2.95 (steps 30–49) to
3.39 (steps 160–199).
Training
| init | amayuelas/Qwen3.5-9B-MatRL-MT-SFT |
| algorithm | GRPO-style multi-turn RL, 10 turns / 8 evaluations per episode |
| batch | 128 rollouts per step, 16 per prompt; zero-advantage groups filtered |
| sampling | temperature 1.15, ≤2,048 tokens per turn |
| reward | SUN-marginal ×1.0 (validity, novelty, relaxed stability: equflashv2 MLIP, 20 steps / fmax 0.1), group diversity ×1.5, band-gap target ×0.5, bulk-modulus target ×0.5 |
| task mix | includes 20% formula-conditioned, 10% unconditional, 10% band-gap-target, 10% bulk-modulus-target prompts |
| steps | 200 |
| optimizer | lr 1e-6, fp32 optimization and gradient reduction |
| seq len | 24,576 (steps 0–159), 32,768 (steps 160–199) |
| trainer | prime-rl |
Trained on 8×A100-40GB: 4 GPUs for the trainer (FSDP + context parallel cp=2, ulysses), 3 for vLLM inference, and 1 for the reward models.
The optimizer runs in fp32 because bf16 does not work at this learning rate. At lr 1e-6 each Adam update is smaller than the spacing between adjacent bf16 values and rounds away. An earlier 200-step attempt in bf16 finished identical to its init, with no errors.
Training caveats
- Seven segments with weights-only restarts. The 200 steps ran as 0–30, 30–50, 50–60, 60–70, 70–90, 90–160 and 160–200. Segments ended on an env-worker death, a vLLM timeout, and three host OOMs, later traced to an unbounded cache in the environment workers. This model class cannot resume from a DCP checkpoint because there is no optimizer state for the vision tower. Each segment therefore restarted from the previous weights with a fresh Adam state.
- Truncation. At 24,576 tokens, the share of episodes clipped at the trainer
window climbed from 1.6% to 38%. The final 40 steps were re-run from step
160, the last low-truncation checkpoint (7.8%), at 32,768 tokens. Truncation
there averaged 14.7% and peaked at 26.6%. A clipped episode's ending, including
the
submitturn, contributes no gradient. - Reward components. Unlike the 4B release, whose bulk-modulus reward never computed, every component here was live. Before each launch, a preflight ran a real relaxation, a real band-gap prediction and the elastic-fit import. Over the last two segments, roughly 100 band-gap requests failed with transient server disconnects and scored no band-gap term.
Precision
Tensors are stored in bf16, and config.json declares bfloat16. The trainer
writes its fp32 optimization dtype into the config, and that value was
corrected for this upload so that dtype="auto" and vLLM do not load the model
at twice the memory.
Important: thinking channel
Reasoning lives in the native thinking channel, preserved across tool calls.
Do not sample with enable_thinking=false, which turns off the behavior this
model was trained to use.
Serving
Includes preprocessor_config.json / video_preprocessor_config.json. vLLM
requires them for this VL-capable model class even for text-only serving.