UnifoLM-VLA — G1 Dex3 Block-Stacking (Real + Synthetic Mix)
Full finetune (VLM backbone not frozen) of UnifoLM-VLA on the Unitree G1 + Dex3-hand block-stacking task, trained on a mix of real and simulation-generated (synthetic) demonstrations.
Companion model trained on real data only: Fichtl00/Unifolm-VLA_real.
Training data
Mixed via data_mix with a 3:1 relative sampling weight (~75% real / ~25% synthetic per epoch):
- Real: unitreerobotics/G1_Dex3_BlockStacking_Dataset
- Synthetic: Fichtl00/Cube_Stacking_synth_jointspace (Isaac Lab simulation, joint-space state/action matching the real dataset's 28-dim layout: 14 arm joints + 14 Dex3 hand joints per side, no waist)
Training setup
- Base VLM: unitreerobotics/UnifoLM-VLM-Base (Qwen2.5-VL)
- Full finetuning (backbone unfrozen) — the reason this ran on KISSKI HPC (4×A100 80GB) rather than locally
- DeepSpeed ZeRO Stage 2, bf16
- Global batch size 24 (per-device 6 × 4 GPUs)
- 30,000 steps, learning rate 4e-5
- Action/state: 28-dim joint space, action horizon 16
Final loss: action_dit_loss = 0.0076, mse_score = 0.0010 (step 30000).
Checkpoints
final_model/ plus intermediate checkpoints every 2000 steps (checkpoints/steps_2000 … steps_30000) for
checkpoint selection / comparison against the real-only run.