Video2VLA RoboTwin-15 Ground-Truth Latent-Slice IDM
A compact inverse-dynamics action head trained on frozen ground-truth HunyuanVideo 1.5 causal-VAE latents from 15 RoboTwin tasks. This is an action model, not a video generator.
Model
- Input: BF16/FP32 latent tensor
[B, 32, 6, H, W](training cache usedH=19,W=38). - Temporal layout: two context latent slices followed by four future latent slices.
- Output: normalized joint targets
[B, 24, 14]. - Architecture: the same 4.408M-parameter spatial decoder is applied independently to every latent slice; each slice predicts four actions.
- No proprioception input.
The training cache was created before Hunyuan posttraining and contains clean VAE encodings. This run did not use generated inference latents, a posttrained LoRA, text encoders, or online VAE encoding. Training added independent flow-style noise sampled uniformly from [0.0, 0.2]; validation used fixed noise 0.1.
Training
- Dataset: 15-task RoboTwin randomized-500 wide corpus.
- Usable indexed windows: 69,704 train / 8,821 validation.
- Steps: 100,000.
- Effective batch: 4 on one NVIDIA GH200.
- Optimizer: AdamW, LR
3e-4, weight decay0.05, 500 warmup steps. - Runtime: BF16; fused optimizer.
- Source commit:
d76f4a79a8a7f38f3e280dcc9d112d7c3592f25b. - Weights & Biases run
Validation
| Checkpoint/point | MSE | MAE |
|---|---|---|
| Best logged point (step 99,000; no checkpoint saved at this step) | 0.011119 | 0.058417 |
| Published final checkpoint (step 100,000) | 0.012537 | 0.060717 |
Metrics are computed on normalized joint targets. They should not be interpreted as task success rates.
Files
checkpoint_100000.pt: model, optimizer, scheduler, resolved config, and step; suitable for resuming with Video2VLA.checkpoint_030000.pt: earlier 30k checkpoint retained for comparison.model.safetensors: model state only.config.yaml: source training configuration.resolved_config.json: fully resolved configuration embedded in the checkpoint.action_stats.json: joint-action normalization statistics.training_metadata.json: provenance, metrics, and SHA-256 hashes.training_code/: relevant source snapshot used by this cache-backed run.
Loading the model-only weights
from safetensors.torch import load_file
from video2vla.action_heads import LatentSliceHead
model = LatentSliceHead(
latent_channels=32,
action_dim=14,
hidden_channels=128,
spatial_pool=8,
mlp_dim=512,
actions_per_latent=4,
dropout=0.05,
)
model.load_state_dict(load_file("model.safetensors"))
model.eval()
The outputs are normalized joint positions; use action_stats.json to invert z-score normalization.
Additional mixed-latent checkpoint
The repository also contains the newer 60% clean / 40% generated-latent IDM, trained for 100,000 steps. It is stored separately so checkpoint_100000.pt at the repository root continues to identify the original clean-only model.