VLAct QwenOFT Qwen3-VL-4B for RoboDojo (100K Fine-tuned)
This repository contains a RoboDojo-fine-tuned StarVLA QwenOFT policy.
It was fine-tuned for 100,000 steps on the 35-task RoboDojo LeRobot v2.1
mixture using 2 nodes x 8 GPUs. The uploaded checkpoint is the exact
steps_100000_pytorch_model.pt used for the evaluation reported below. It is
a complete framework state dict, not a LoRA or adapter-only release.
Initialization
VLAct pretrained describes only the initialization source. RoboDojo
fine-tuning started from the 100K-step 0706 multi-embodiment, three-head VLAct
pretraining run. The checkpoint released here is the resulting RoboDojo
fine-tune; it is not the raw VLAct pretraining checkpoint.
Model details
| Item | Value |
|---|---|
| Framework | StarVLA QwenOFT |
| VLM backbone | StarVLA/Qwen3-VL-4B-Instruct-Action |
| Source initializer | 0706_agilex_franka_5data_manualvel_balance33_66_50_vlm3_qwen3OFT_GR00T_PI_xrobot_3head_agilex_franka, step 100K |
| Initialization setting | reload_modules: qwen_vl_interface; training starts from step 0 (is_resume: false) |
| Fine-tuning dataset | robodojo_v21_all_h50_q99, 35 training tasks |
| Observation | Three RGB cameras at 224 x 224 plus 14D proprioceptive state |
| Action | 14D absolute joint position (abs_qpos) |
| Action horizon | 50 |
| Action model | QwenOFT diffusion model, 16 layers |
| Repeated diffusion steps | 8 |
| Inference diffusion steps | 4 |
| Evaluation execution horizon | 16 |
| Fine-tuning checkpoint | 100,000 steps |
| Fine-tuning hardware | 2 nodes x 8 GPUs |
| Batch size | 16 per GPU, global batch 256 |
The original run used the action-extended Qwen3-VL snapshot at revision
8c99a9a3db659019e6e785add928fcea8e14a18a. The packaged config.yaml
uses the public Hub model ID so that the framework can reconstruct the model.
training_config.original.yaml preserves the exact training configuration and
local provenance paths used by the run.
Training details
| Setting | Value |
|---|---|
| Training seed | 42 |
| Frozen modules | None |
| VLA / VLM loss scale | 1.0 / 0.0 |
| Gradient accumulation | 1 |
| Gradient checkpointing | Enabled |
| Optimizer | AdamW, betas (0.9, 0.95), epsilon 1e-8 |
| VLM learning rate | 1e-5 |
| Qwen-VL interface learning rate | 1e-5 |
| Action-model learning rate | 1e-4 |
| Schedule | Cosine, 5,000 warmup steps, minimum LR 5e-7 |
RoboDojo results
Values are shown as success rate (SR, %) and score; higher is better. Group values are unweighted means over their tasks, and Average is the unweighted mean over all 42 tasks.
| Evaluation group | SR (%) | Score |
|---|---|---|
| Average | 8.17 | 11.28 |
| Generalization | 4.72 | 8.18 |
| Precision | 18.33 | 23.44 |
| Long-Horizon | 15.42 | 21.29 |
| Memory | 0.56 | 0.67 |
| Open | 1.67 | 1.71 |
Evaluation protocol
- 42 tasks, seeds 0/1/2, 10 episodes per task and seed: 1,260 episodes total.
- Each task therefore has 30 evaluated episodes.
- Each Generalization task uses 5 standard and 5 randomized episodes per seed, preserving the official standard/random split at one-fifth scale.
- All other tasks use 10 episodes per seed.
- Evaluation uses state (
include_state=true),arx_x5normalization, three camera inputs, and executes 16 actions before requesting the next chunk. - Task SR is successful episodes divided by 30. Task score is the mean episode score multiplied by 100.
This is a scaled local reproduction protocol. It is not the official 50-episode-per-task / 2,100-episode protocol, and the values should not be presented as a verified official leaderboard submission.
Per-task results
| Group / task | SR (%) | Score |
|---|---|---|
| Generalization | 4.72 | 8.18 |
| stack_bowls | 26.67 | 31.67 |
| push_T | 0.00 | 0.00 |
| pack_objects_into_box | 0.00 | 5.33 |
| fold_clothes | 0.00 | 4.67 |
| hang_mugs | 3.33 | 7.83 |
| sweep_blocks | 0.00 | 0.00 |
| pour_liquid_into_cup | 6.67 | 6.67 |
| make_toast | 0.00 | 5.83 |
| arrange_largest_number | 0.00 | 2.17 |
| sort_nesting_dolls_by_size | 3.33 | 3.33 |
| store_laptop_and_headphones | 3.33 | 11.33 |
| stack_blocks | 13.33 | 19.33 |
| Precision | 18.33 | 23.44 |
| fasten_screws | 0.00 | 5.67 |
| plug_in_charger | 10.00 | 10.00 |
| insert_tubes | 56.67 | 66.00 |
| pour_balls_into_vase | 16.67 | 16.67 |
| play_Xylophone | 0.00 | 0.00 |
| deposit_coin | 10.00 | 14.67 |
| insert_key | 0.00 | 14.50 |
| build_tower | 53.33 | 60.00 |
| Long-Horizon | 15.42 | 21.29 |
| put_bottles_into_dustbin | 63.33 | 71.33 |
| fill_pen_holder | 0.00 | 12.00 |
| classify_objects | 0.00 | 4.83 |
| play_tic_tac_toe | 0.00 | 4.33 |
| fill_egg_holder | 0.00 | 2.00 |
| organize_table | 3.33 | 19.17 |
| make_kong | 56.67 | 56.67 |
| play_stacking_toy | 0.00 | 0.00 |
| Memory | 0.56 | 0.67 |
| cover_blocks | 0.00 | 0.00 |
| match_and_pick_from_conveyor | 3.33 | 3.33 |
| swap_blocks | 0.00 | 0.00 |
| swap_T | 0.00 | 0.00 |
| press_by_number | 0.00 | 0.00 |
| imitate_sorting_sequence | 0.00 | 0.67 |
| Open | 1.67 | 1.71 |
| align_blocks | 0.00 | 0.00 |
| general_pickup | 13.33 | 13.33 |
| stack_blocks_by_language | 0.00 | 0.00 |
| solve_equation | 0.00 | 0.00 |
| classify_objects_by_language | 0.00 | 0.33 |
| pick_from_conveyor_by_image | 0.00 | 0.00 |
| store_tools_in_toolbox | 0.00 | 0.00 |
| pour_by_language | 0.00 | 0.00 |
Machine-readable values and protocol metadata are in
evaluation_results.json.
Files
README.md
config.yaml
dataset_statistics.json
evaluation_results.json
summary.jsonl
training_config.original.yaml
checkpoints/
└── steps_100000_pytorch_model.pt
config.yaml and dataset_statistics.json must remain two directory levels
above the checkpoint file; StarVLA resolves both relative to the checkpoint.
Inference
Download the repository, then start the StarVLA model server:
huggingface-cli download JasonYang66/VLAct-Qwen3vl4b-OFT-RoboDojo \
--local-dir VLAct-Qwen3vl4b-OFT-RoboDojo
CKPT=VLAct-Qwen3vl4b-OFT-RoboDojo/checkpoints/steps_100000_pytorch_model.pt
python deployment/model_server/server_policy.py \
--ckpt_path "$CKPT" \
--port 57700 \
--use_bf16
RoboDojo evaluation must match the training-time input contract:
export STARVLA_CKPT_PATH="$CKPT"
export STARVLA_INCLUDE_STATE=True
export STARVLA_UNNORM_KEY=arx_x5
export STARVLA_EXECUTE_HORIZON=16
Changing state usage, camera ordering, normalization statistics, action order, or execution horizon can materially change the result.
Intended use
This checkpoint is intended for research on RoboDojo with the ARX X5 dual-arm embodiment. Performance under different camera calibration, state/action ordering, normalization, robot embodiments, or real hardware has not been established.