RoboTok — A Scalable Data Engine for Internet Demonstration Video Retrieval and Dexterous Manipulation Learning
🌐 Project website: rice-robotpi-lab.github.io/RoboTok
Released checkpoints and evaluation keypoints for RoboTok, a model that retrieves web video clips by 3D hand-motion similarity. Similarity is defined by DTW over torso-relative 3D hand keypoints; the encoder is trained to reproduce that DTW ranking in a fast embedding space.
Training and evaluation code is in the accompanying source release.
Files
| File | Size | Description |
|---|---|---|
models/best_abs_retrieval_model.pt | 3.8 MB | Retrieval encoder. Cross-attention head over [T, 126] hand-trajectory features (21 joints x 3 coords x 2 hands, T_max = 42): 1 learned query token, 256-d input projection, 1 cross-attention layer (4 heads, sinusoidal PE), 2-layer MLP to a 256-d embedding. DTW design abs_21j_coords. |
models/best_abs_retrieval_model.yaml | 2 KB | Minimal config to reload the encoder for inference. |
models/body_pose_est.pt | 9.9 MB | Vector-neuron torso/body-frame estimator: 4-layer rotation-equivariant transformer mapping two-hand trajectories to a torso frame, with separate rotation and translation heads. |
eval_data/torso_relative_clip_keypoints.pt | 6.5 GB | Torso-relative 3D hand keypoints per clip. Each entry has video_number, node_number, node_uid, keypoints_per_frame (kpts_2d, kpts_3d), and infilled / depth_grounded flags. |
Loading
import torch
ckpt = torch.load("models/best_abs_retrieval_model.pt", map_location="cpu", weights_only=True)
ckpt["head_state_dict"] # encoder weights
ckpt["config"] # full training configuration
vn = torch.load("models/body_pose_est.pt", map_location="cpu", weights_only=True)
vn["model"] # torso estimator weights
Citation
@article{qian2026robotok,
title = {RoboTok: A Scalable Data Engine for Internet
Demonstration Video Retrieval and Dexterous Manipulation
Learning},
author = {Qian, Howard and Chen, Yiting and Xie, Yunfei and
Ren, Kejia and Chanrungmaneekul, Podshara and Wang, Gaotian and
Wen, Bowen and Wei, Chen and Hang, Kaiyu},
journal = {arXiv preprint arXiv:2609.03199},
year = {2026}
}
License
FAIR Noncommercial Research License v1 (see LICENSE-Action100M). Noncommercial research only.
The released checkpoints (models/*.pt) and evaluation keypoints
(eval_data/torso_relative_clip_keypoints.pt) are derivative works of
Action100M (Meta FAIR) clips and are governed by that license. It covers
trained model weights as "Research Materials", and restricts both those
materials and any outputs or results obtained from them to noncommercial
research use. If you publish results obtained using these materials, the
license requires you to acknowledge that use.
MIT (LICENSE) covers only models/best_abs_retrieval_model.yaml
and the accompanying source release.
The MANO / SMPL-H body models required by parts of the pipeline are not included and remain under their own MPI-IS license terms — register at https://mano.is.tue.mpg.de to obtain them.