Qwen3-4B Chemistry SDPO
This repository contains Chemistry fine-tuned Qwen3-4B checkpoints from the local SciKnowEval-style generalization setup.
- Root checkpoint: final
global_step_100merged to Hugging Face safetensors. best_avg16/: checkpoint with the highest validationavg@16during training, merged to Hugging Face safetensors.
Checkpoints
| Checkpoint | Source step | Validation avg@16 | best@16 | maj@16 |
|---|---|---|---|---|
| Root final | 100 | 0.720536 | 0.731567 | 0.721367 |
best_avg16/ | 20 | 0.766369 | 0.822871 | 0.779486 |
Training Run
qwen3gen-chemistry-SDPO-Qwen-Qwen3-4B-mbs32-ema0.05-train256-rollout8-lr1e-5-vllm0.8
W&B run: https://wandb.ai/seongryongjung-chung-ang-university/qwen3-generalization-batch256/runs/1qazekk3
Base Model
- Base model:
Qwen/Qwen3-4B - Fine-tuning type: full-parameter FSDP RL training
- Dataset:
datasets/sciknoweval/chemistry - Train split: 1,890 examples
- Validation split: 210 examples
Method
- Method: SDPO
- Config:
sdpo - Policy loss mode:
sdpo - Reward: local SciKnowEval multiple-choice reward checker
- Rollout correction: token-level importance sampling, threshold 2.0
Hyperparameters
| Field | Value |
|---|---|
| Base model | Qwen/Qwen3-4B |
| Training steps | 100 |
| Train batch size | 256 |
| Rollouts per prompt | 8 |
| Generations per step | 2048 |
| PPO mini batch size | 32 |
| Learning rate | 1e-5 |
| LR warmup steps | 10 |
| Weight decay | 0.01 |
| Grad clip | 1.0 |
| Max prompt length | 2048 |
| Max response length | 8192 |
| Max model length | 10240 |
| Train temperature | 1.0 |
| Train top_p | 1.0 |
| Validation generations | 16 |
| Validation temperature | 0.6 |
| Validation top_p | 0.95 |
| vLLM GPU memory utilization | 0.8 |
| GPUs | 8 x NVIDIA H200 |
| Save frequency | every 10 steps |
| Validation frequency | every 10 steps |
| Distillation top-k | 100 |
| SDPO alpha | 0.5 |
| Teacher update rate | 0.05 |
| Distillation IS clip | 2.0 |
| Max reprompt length | 10240 |
Metrics
CSV files:
| Metric | Value |
|---|---|
| Final training step | 100 |
Final critic/score/mean | 0.845215 |
Final critic/rewards/mean | 0.845215 |
Final validation avg@16 | 0.720536 |
Peak validation avg@16 | 0.766369 |
| Peak validation step | 20 |
Loading
Root final checkpoint:
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("SeongryongJung/Qwen3-4B-Chemistry-SDPO")
tokenizer = AutoTokenizer.from_pretrained("SeongryongJung/Qwen3-4B-Chemistry-SDPO")
Best avg@16 checkpoint:
model = AutoModelForCausalLM.from_pretrained("SeongryongJung/Qwen3-4B-Chemistry-SDPO", subfolder="best_avg16")
tokenizer = AutoTokenizer.from_pretrained("SeongryongJung/Qwen3-4B-Chemistry-SDPO", subfolder="best_avg16")
Intended Use
This model is intended for research on RL fine-tuning and self-distillation behavior on science/generalization tasks. It has not been broadly safety evaluated for production use.
Limitations
The reported scores are training-time and validation-time metrics from the local experimental setup. They should not be interpreted as broad benchmark results without independent evaluation.