RPT-VLA
This repository contains a Hugging Face-style export of RPT-VLA, a diffusion-action vision-language-action model for SimplerEnv evaluation. The model takes a language instruction and an RGB observation as input, then predicts 7-DoF robot actions of the form (x, y, z, roll, pitch, yaw, gripper).
The checkpoint is packaged for offline local loading: model weights, tokenizer files, configuration files, and dataset statistics are stored in a single top-level directory.
Model Summary
- Model type: Vision-language-action policy with a diffusion action decoder
- Base VLM:
prism-dinosiglip-224px+7b - Vision backbone: DINOv2 + SigLIP fused vision backbone
- Language model: Llama-2 7B backbone
- Action decoder: DiT-B diffusion action head
- Action dimension: 7
- Future action window size: 15
- Past action window size: 6
- Repeated diffusion steps: 4
- Supported normalization keys:
fractal20220817_data,bridge_dataset - Checkpoint format: flat Hugging Face-style
safetensorsshards
Files
config.json: model configuration used by the local diffusion-action loaderconfig.yaml: original run configuration in YAML formdataset_statistics.json: action normalization statistics for supported evaluation datasetsmodel.safetensors.index.json: index mapping parameters to safetensors shardsmodel-*.safetensors: sharded model weightstokenizer.model,tokenizer.json,tokenizer_config.json,special_tokens_map.json,added_tokens.json: tokenizer filesllama2_7b_config.json: local Llama-2 backbone configuration used for offline model constructionoffline_backbones.py: offline construction helper for local evaluation environmentsmodel_format.json: lightweight metadata describing the exported checkpoint layout
Uses
This model is intended for SimplerEnv robot policy evaluation and related fine-tuning workflows. It can be evaluated on task families whose action normalization statistics are included in dataset_statistics.json.
For execution, actions must be un-normalized with the correct unnorm_key:
fractal20220817_datafor Google Robot-style tasksbridge_datasetfor WidowX / Bridge-style tasks
Using the wrong normalization key can produce invalid actions and unreliable evaluation results.
Getting Started
Use the accompanying project evaluation scripts and pass this directory as --model-dir.
Example for Google Robot-style tasks:
python scripts/eval.py \
--robot google_robot \
--model-dir /path/to/model \
--simpler-env-root /path/to/simpler_env \
--log-root /path/to/output/google_robot \
--gpu 0 \
--families ALL \
--protocol both
Example for WidowX / Bridge-style tasks:
python scripts/eval.py \
--robot widowx_robot \
--model-dir /path/to/model \
--simpler-env-root /path/to/simpler_env \
--log-root /path/to/output/widowx_robot \
--gpu 0
Evaluation Arguments
The evaluation entry is scripts/eval.py. The main configurable arguments are:
| Argument | Default | Choices / Type | Description |
|---|---|---|---|
--model-dir | ../model | path | Path to this HF-style RPT-VLA model directory or another compatible model directory. |
--simpler-env-root | SIMPLERENV_ROOT or ./SimplerEnv | path | Path to the local SimplerEnv checkout. |
--log-root | ../logs/eval | path | Output directory for progress.jsonl, progress_summary.txt, and videos/. |
--gpu | 0 | string / int | GPU id passed as CUDA_VISIBLE_DEVICES to the evaluation process. |
--robot | google_robot | google_robot, widowx_robot | Selects the evaluation benchmark. |
--families | pick_coke_can,move_near,drawer | comma-separated list or ALL | Google Robot task families to evaluate. Used only when --robot google_robot. |
--protocol | both | variant, visual_matching, both | Google Robot evaluation protocol subset. Used only when --robot google_robot. |
--dry-run | disabled | flag | Print the commands that would run without starting evaluation. |
--stop-on-error | disabled | flag | Stop after the first failed task subprocess. |
--append-progress | disabled | flag | Append to an existing output directory instead of clearing previous progress files. |
--save-videos | true | true / false | Save rollout videos. |
--save-action-images | true | true / false | Save action visualization images. |
--print-step-info | false | true / false | Print per-step simulator info to stdout. |
--use-bf16 | true | true / false | Use bfloat16 for the VLM backbone. |
--cfg-scale | 1.5 | float | Classifier-free guidance scale for diffusion action sampling. |
--num-ddim-steps | 10 | int | Number of DDIM sampling steps. |
--use-ddim | true | true / false | Use DDIM sampling for the diffusion action decoder. |
--action-ensemble | true | true / false | Enable temporal action ensembling. |
--action-ensemble-horizon | robot default | int or unset | Override the default ensemble horizon. Defaults to 2 for Google Robot and 7 for WidowX. |
--adaptive-ensemble-alpha | 0.1 | float | Weighting factor used by adaptive action ensembling. |
--future-action-window-size | model config | int or unset | Override the model future action window size. Leave unset for normal evaluation. |
--past-action-window-size | model config | int or unset | Override the model past action window size. Leave unset for normal evaluation. |
--unnorm-key | robot default | string or unset | Override action normalization statistics. Defaults to fractal20220817_data for Google Robot and bridge_dataset for WidowX. |
Google Robot supports these task families:
pick_coke_canmove_neardrawerput_in_drawer
WidowX evaluation currently runs the bundled Bridge task set and does not use --families or --protocol.
Each evaluation output directory contains:
progress.jsonl: rollout-level resultsprogress_summary.txt: aggregate success ratesvideos/: rollout videos and action visualizations, if enabled
Limitations
The model is designed for the robot embodiments and evaluation setups represented by its normalization statistics and training data. It should not be assumed to transfer directly to unseen robot embodiments or substantially different camera/action conventions without additional validation or fine-tuning.