π0.5 LIBERO — reverse-direction manipulation, rotation-clipped data (10,000 steps)

An action-expert fine-tune of siruku6/pi05_full_runpod on a mixture of lerobot/libero_plus and ~49.5 % synthetic reverse-direction episodes we generated in LIBERO simulation.

The target skill is the reverse of what LIBERO demonstrates: the distributed demos put objects onto shelves, plates and trays, and the tasks we needed take them off again (take the red mug off the left plate and put it on the table). Only one of 2,066 instruction strings in the distributed training data is a "take … off / out of" task, so we synthesised the missing direction with a scripted controller.

This is a published negative result. The data fix this run was built to test did land — the policy now issues wrist-rotation commands of the same magnitude as the data, instead of shrinking them toward the mean. But the policy still does not grasp. Across 20 scripted evaluation rollouts it lifted the target object zero times; the two runs where the object rose more than 30 mm were the arm knocking the scene over (other objects displaced by 279 mm and 299 mm). The weights and the training log are here so the comparison is reproducible.

Lineage

Contents

Five checkpoints, saved every 2,000 steps, in LeRobot's on-disk layout, plus the full training log.

checkpoints/002000/pretrained_model/   config.json, model.safetensors (9,354,050,752 B),
                                       policy_preprocessor.json + normalizer safetensors,
                                       policy_postprocessor.json + unnormalizer safetensors,
                                       train_config.json
checkpoints/002000/training_state/     optimizer_state.safetensors (2,195,136,504 B),
                                       optimizer_param_groups.json, scheduler_state.json,
                                       rng_state.safetensors, training_step.json
... 004000/ 006000/ 008000/ 010000/
pi05_t3rev_v2.log

The vlm/ tokenizer directory is not included. A policy server that constructs the tokenizer offline needs it, 17 MB. See Getting the vlm/ tokenizer below.

Training setup

The data

The reverse episodes cover three tasks (yellow book off a shelf, alphabet soup out of a tray, red mug off a plate). They are not time-reversed playback. Each one is an ordinary forward-in-time simulator rollout in six phases — approach with the gripper open, close in place, lift, follow a return path, place, release — recorded at 20 fps with images, actions and states all advancing in time. Only the return path's reference trajectory is read backwards out of a distributed forward demo.

What changed versus the previous attempt

An earlier run used a first version of this data in which the scripted controller's per-step wrist rotation command saturated at the ±1 action clip, while the distributed demos stay inside ±0.375. 15.2 % of frames exceeded 10 σ, and 52.6 % of those were in the first 10 % of each episode — exactly the approach phase. This version rate-limits rotation to ±0.375, and 0.0 % of frames exceed 10 σ.

The effect on the policy is measurable, and it is the one the fix predicted:

Training loss also comes down: 0.41–0.44 → 0.346 at the same 8,700 steps, same recipe, same mixture ratio.

Evaluation — what still fails

Measured in LIBERO simulation on the four reverse-direction evaluation tasks (three of which have reverse training data), 300 steps per rollout.

Object selection is not the bottleneck. Measured without rollouts — integrate the 50-step action chunk into a heading, compare it against every object in the scene — the sibling policy trained on the earlier version of this data aims at the correct object on two of the three tasks (cosine +0.97 and +0.76, against the strongest competitor). The failure is downstream: started from a training episode's own initial state, the policy leaves the recorded path within the first 10–20 % of it and never comes back within 8 cm, where two valid recorded paths from the same initial state differ by only 1–20 mm.

Raising the number of executed steps per plan from 16 to 50 makes it worse on every measure, so plan truncation is not the cause either.

Getting the vlm/ tokenizer

vlm/ is the PaliGemma tokenizer, unmodified. Its canonical source is google/paligemma-3b-pt-224. That repository is gated: sign in, open the model page, accept the Gemma license and wait for access to be granted, then authenticate locally with hf auth login before the download will work.

hf download google/paligemma-3b-pt-224 \
    tokenizer.json tokenizer_config.json special_tokens_map.json added_tokens.json \
    --local-dir <checkpoint>/vlm

Those four files, about 17 MB, are byte-for-byte what vlm/ holds; tokenizer.json has md5 e00c4623084b205875ed738d5685f24b. Put them in place as real files, not symlinks.

Note that the tokenizer is not in lerobot/pi05_libero_base — that repository ships only model.safetensors, config.json and the two pre/post-processor JSON files.

Check that tokenizer_config.json says "tokenizer_class": "GemmaTokenizer". A directory containing merges.txt and vocab.json is a SmolVLM2 tokenizer, not this one; the policy will run and silently fail to follow instructions.

Use

hf download siruku6/pi05_t3rev_v2 --include 'checkpoints/010000/pretrained_model/*' \
    --local-dir ./t3rev_v2
hf download google/paligemma-3b-pt-224 \
    tokenizer.json tokenizer_config.json special_tokens_map.json added_tokens.json \
    --local-dir ./t3rev_v2/checkpoints/010000/pretrained_model/vlm

Then point LeRobot at the local directory:

lerobot-train --policy.type=pi05 \
    --policy.pretrained_path=./t3rev_v2/checkpoints/010000/pretrained_model ...

Intended use and limitations

Research artifact. Trained and evaluated only in LIBERO simulation with a simulated Franka Panda — there is no real-robot validation, and nothing here should be run on physical hardware without your own safety review. Performance outside the LIBERO task and camera setup is unknown. The policy does not reliably grasp; see the evaluation section before building on it.

License

These weights are a Model Derivative of Gemma (via PaliGemma inside π0.5) and are released under the Gemma Terms of Use. Use is also subject to the Gemma Prohibited Use Policy. See NOTICE for the third-party attributions that come with the base model, the data and the training code.