ACE-Ego-Hand (safetensors)
Inference weights of ACE-Ego-Hand (Liu et al., 2026, arXiv:2608.20308), repackaged as safetensors for the reimplementation at https://github.com/julien-blanchon/ACE-Ego-Hand-Pipeline. The model recovers bimanual 3D hand motion (MANO pose, shape and camera-space translation per frame) from egocentric RGB video by reading the hidden state of a Wan2.2 video diffusion transformer.
Nothing here is retrained. Every tensor is a byte-for-byte copy of its source, renamed and, for the transformer, pruned to the blocks the model actually runs.
Files
| path | what | source |
|---|---|---|
dit/ | Wan2.2-5B trunk, blocks 0-15 only (block 15 is the feature tap), bf16 | alibaba-pai/Wan2.2-Fun-5B-Control |
vae/ | Wan2.2 VAE encoder (48 latent channels, 16x / 4x), bf16 | Wan-AI/Wan2.2-TI2V-5B-Diffusers |
taehv/ | TAEHV taew2_2 encoder, the tiny approximation of the VAE, fp32 | madebyollin/taehv |
caption_embedding.safetensors | the one constant umT5-XXL caption embedding the trunk cross-attends to, fp32 | computed from Wan-AI/Wan2.2-TI2V-5B-Diffusers |
detector/ | YOLOv8-m hand detector (26 M, fp32), the second opinion behind InferenceConfig.verify="detector"; CC-BY-NC-4.0 weights, see detector/README.md | rolpotamias/WiLoR detector.pt |
kfree/, k/ | per variant: backbone_delta.safetensors (LoRA rank 64 on blocks 0-15 + the trained patch embedding, fp32), projector/ (the readout: alternating encoder, heads, ray map), config.json | acerobotics2025/ACE-Ego-Hand |
mano/ | both MANO hands as tensors (template, blend shapes, regressor, skinning, faces), fp32 | MANO |
config.json | variants, encoders, encode width, window length |
Each dit/, vae/, taehv/, mano/ and */projector/ folder is a config.json + model.safetensors
pair loadable with huggingface_hub.PyTorchModelHubMixin (from_pretrained(folder)), and has a
README saying what it is.
kfree needs no camera calibration (the model predicts a ray field and fits its own pinhole);
k takes the intrinsics. Both are pinhole-only.
Usage
uv run --with "ace-ego-hand @ git+https://github.com/julien-blanchon/ACE-Ego-Hand-Pipeline" \
ace-ego-hand infer https://huggingface.co/spaces/blanchon/ACE-Ego-Hand/resolve/main/examples/holoassist_coffee_machine.mp4 --render
That downloads the weights from this repository on first use, writes holoassist_coffee_machine.hands.parquet
(one row per frame and hand) and the rendered holoassist_coffee_machine.hands.mp4 in the working
directory. Any local file or URL works in place of the example.
See the repository README for the Python API, the LeRobot annotation script and the demo.
Variants
The int8 branch of this repository holds the same weights with the trunk (dit/) quantized by
torchao to int8 weight-only: the 160 block linears keep int8
storage and a per-row scale and are dequantized to bf16 at the matmul, everything else (patch,
time and text embeddings, norms, the LoRA adapters of the variant folders) is unchanged. It
halves the trunk on disk (5.42 -> 2.80 GB) and cuts peak GPU memory by about 2.4 GiB at the same
speed on CUDA, for a joint error of 0.45-0.8 mm median against the bf16 trunk (whose run-to-run
noise floor is 0.35 mm). Only dit/ differs on that branch; the loader fetches it from there
and every other component from main. Use it with --model.trunk-precision int8
(trunk_precision="int8" in ModelConfig), which needs the int8 dependency extra
(ace-ego-hand[int8], i.e. torchao>=0.16); see dit/README.md on the branch for the file
layout.
Licence
The ACE-Ego-Hand weights are CC BY-NC 4.0 (they derive from MANO and research-licensed datasets; no commercial use). The Wan2.2 weights are Apache 2.0. TAEHV is MIT. MANO is under its own licence from the Max Planck Institute.
Citation
@misc{liu2026aceegohand,
title={ACE-Ego-Hand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Motion Recovery},
author={Yufei Liu and Xixi Wang and Hao Li and Ganlong Zhao and Kaitong Cai and Chengkai Jin and Chunxiao Liu and Jianbo Liu and Siyuan Huang and Xingang Pan and Hongsheng Li},
year={2026},
eprint={2608.20308},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2608.20308},
}