StarVLA-OFT — G1 + Inspire piston pick-and-place

Full fine-tune of StarVLA (QwenOFT framework: Qwen3-VL-2B backbone + MLP L1-regression action head) on birbirll/g1-inspire-piston-pick-place (102 success episodes, LeRobot v2.1). Vision + language only — no state input (see recipe notes). Closed-loop stable in IsaacLab.

GR00T-N1.6 baseline trained on the same dataset: birbirll/g1-inspire-piston-n16.

Contents

• final_model/pytorch_model.pt — bf16 weights (whole model, self-contained)
• config.yaml / config.full.yaml / dataset_statistics.json — the run layout StarVLA's policy_wrapper expects (config + stats one level above the weights)
• starvla_files/ — drop-ins for your StarVLA checkout:
  • data_config.py → .../train_files/data_registry/ (registers the unitree_g1_piston embodiment + unitree_g1_piston_pnp mixture this checkpoint references)
  • starvla_qwenoft_g1_piston_v2.yaml — the exact training config
  • policy_norm_processor_PATCHED.py — required to serve this checkpoint: upstream's PolicyNormProcessor crashes (Video key ego_view not found) on any data config whose transform includes video augmentation, as this one does. Replace deployment/model_server/policy_norm_processor.py.

Point framework.qwenvl.base_vlm in config.yaml at your local Qwen/Qwen3-VL-2B-Instruct (tokenizer/processor are loaded from there; the fine-tuned weights come from this repo's .pt).

Recipe (single RTX 4090, ~2h15m; transfers to bigger GPUs)

Action horizon 30, action_mode: abs, image 224, color jitter (0.3/0.4/0.5/0.08), effective batch 32 (micro-batch 8 × grad-accum 4), 10k steps, per-module LRs (backbone 2e-5 / VL interface 1e-5 / head 1e-4), bnb PagedAdamW8bit, gradient checkpointing, no DeepSpeed (single GPU).

Three hard-won gotchas if you retrain:

1. include_state: false on purpose. With state enabled on a stationary task, the model regresses actions from proprioception and ignores the camera (looks converged, fails closed-loop). StarVLA has no state-dropout to mitigate this — leave state off, or wire the dormant StateActionDropout transform yourself.
2. Never use gradient_accumulation_steps > 1 with DeepSpeed in train_starvla.py — the trainer steps the LR scheduler per micro-batch, silently compressing your schedule. On a single GPU, drop DeepSpeed entirely (ZeRO shards nothing across one rank) and accumulation via ACCELERATE_GRADIENT_ACCUMULATION_STEPS works correctly.
3. Full fine-tune, don't freeze the VLM. QwenOFT has no projector — freezing the backbone leaves a 42M head learning manipulation from web-pretrained features that never adapt. We verified a frozen-VLM sibling run converges onto the state shortcut.

Notes

Works with StarVLA's own deployment/model_server/server_policy.py (WebSocket, msgpack-numpy): request {"examples": [{"image": [rgb_hwc_uint8], "lang": "pick up the piston."}], "unnorm_key": "new_embodiment"} → {"data": {"actions": [B, 30, 30]}}, absolute physical units.