pi0.5 — frozen VLM, relative actions, joint space (10-dim motor targets)
Fine-tuned from lerobot/pi05_base on the
949-episode base4 multi-task mixture (900 train / 49 held out, 8 tasks, 3 cameras, 50 Hz),
with the VLM frozen: train_expert_only=true and freeze_vision_encoder=true, so only the
action expert and projections are trained — 693M of 4.14B parameters.
Final checkpoint, step 30000 of 30000. Held-out loss fell monotonically and never turned up, so the last checkpoint is also the best one.
| step | held-out loss |
|---|---|
| 5K | 0.0368 |
| 10K | 0.0327 |
| 15K | 0.0312 |
| 20K | 0.0302 |
| 25K | 0.0297 |
| 30K | 0.0296 |
Why the VLM is frozen
Fully fine-tuning all ~4B parameters on 900 episodes risks overfitting. This arm tests the lerobot pi0.5 guide's lower-capacity variant against the same data and the same action representation. The matched full-fine-tune comparison is being run now and is not quoted here until it exists — an earlier revision of this card carried a comparison figure that traced to a different experiment, and it has been removed rather than restated.
Camera sensitivity
Measured, and low. Holding observation.state fixed and swapping in camera frames from other
held-out episodes, then asking how far the predicted action moved, relative to the natural
cross-episode spread of that action:
| value | |
|---|---|
| arm sensitivity ratio | 0.091 (95% CI 0.075-0.107) |
| gripper ratio | 0.237 — reported separately, never read as arm grounding |
| prediction-error ratio | 0.072 (guardrail: must be <= 0.8) |
| null test | 0.00000 (guardrail: must be <= 0.01) |
| episode clusters | 49, 10,372 rows, evaluator revision 2 |
A ratio near 1.0 means the policy tracks the visual scene; near 0.0 means it ignores the cameras. Both guardrails pass, so this is a genuine negative result rather than an artifact: this policy predicts arm motion largely without reading its cameras, extrapolating proprioception and task identity instead. Sensitivity does grow with prediction lead — 0.007 at 0.0 s to 0.168 at 0.96 s — but stays far below the 0.5 grounding threshold throughout.
The measurement is only interpretable because held-out loss has plateaued (<1% relative improvement across the last two evaluations); a sensitivity number taken mid-training is not.
Exploratory, not confirmatory. One inference seed, 3 anchors and 2 donors per episode. A confirmatory claim needs four inference seeds with common random numbers and three training seeds. Do not read the joint-space and end-effector numbers as a comparison with each other: their denominators differ by orders of magnitude and native ratios are commensurate only within one action representation.
Deploying without the training dataset
train_config.json records an absolute dataset.root from the training machine, and the
l5vel-peng/multitask-relative-h50 view is not published, so a robot can resolve neither. Everything inference needs is
already inside this repo, so the dataset is not required:
- normalisation statistics are baked into
policy_preprocessor_step_3_normalizer_processor.safetensors(120 tensors: mean/std/min/max/quantiles for every feature); action_feature_namesis stored inconfig.json, which is what drives the relative-action mask and keeps the gripper absolute.
Load it directly and pass no dataset metadata:
from lerobot.configs.policies import PreTrainedConfig
from lerobot.policies.factory import get_policy_class, make_pre_post_processors
repo = "maskjp/pi05-relative-joints-frozen-vlm"
cfg = PreTrainedConfig.from_pretrained(repo)
cfg.device = "cuda"
policy = get_policy_class(cfg.type).from_pretrained(repo, config=cfg)
pre, post = make_pre_post_processors(cfg, pretrained_path=repo)
make_policy(cfg, ds_meta=...) is the thing to avoid: when ds_meta is not None, LeRobot
overwrites action_feature_names from the dataset, which is exactly the path that needs the
missing files.
If your deployment insists on dataset metadata, this repo also ships meta/ (~1.8 MB:
info.json, stats.json, tasks.parquet, episodes/). Point dataset.root at a directory
containing it — no data/ or videos/ needed:
huggingface-cli download maskjp/pi05-relative-joints-frozen-vlm --include 'meta/*' --local-dir ./policy_meta
# then use dataset.root=./policy_meta
That info.json carries flat action names. An older grouped form, {"motors": [...]},
degrades to ["motors"] in policies/factory.py, which pads the relative mask all-True and
relativises the gripper against statistics that hold it absolute. The shipped copy is the
migrated one and the publisher refuses to upload a grouped one.
Action representation
Targets are relative: action[t+k] - state[anchor], one anchor per chunk, added back after
inference. The gripper stays absolute (relative_exclude_joints=['gripper']) — it is a
near-binary command, not a pose.
pi0.5 applies the relative conversion before normalisation, so the normaliser needs offset
statistics. This trains against l5vel-peng/multitask-relative-h50, whose meta/stats.json holds quantiles recomputed
over chunk-50 offsets. Absolute statistics would squash every target.
Configuration
pretrained_path lerobot/pi05_base
chunk_size 50 n_action_steps 10 n_obs_steps 1
freeze_vision_encoder true train_expert_only true
gradient_checkpointing false compile_model true dtype bfloat16
optimizer_lr 2.5e-5, warmup 1000, cosine to 2.5e-6 over 30K
norm VISUAL IDENTITY | STATE QUANTILES | ACTION QUANTILES
batch 21/rank x 3 GPUs = 63 effective, DDP, seed 1000
Apache-2.0, inherited from LeRobot.