HTT — Heterogeneous Tactile Transformer
[!IMPORTANT] Moved: model and dataset are now hosted together in a single repo — AllenBi21/HTT-dataset. Please download the checkpoint from there: /home/allenbi/.cache/huggingface/hub/datasets--AllenBi21--HTT-dataset/snapshots/b3d6f0a275dfc949cbe6c100e4ca767e4ad8bf59/htt_4sensors_best.pth. This repo is kept only so existing links keep working.
A multimodal tactile representation model. One shared transformer backbone encodes four different tactile sensors into a common 192-dimensional embedding space, pretrained with masked-autoencoder reconstruction and cross-modal alignment. Feed a raw sensor reading, get a feature vector for any downstream head (classification, force / slip estimation, policy learning).
| Modality | Type | Raw input |
|---|---|---|
gsmini | vision (GelSight Mini) | uint8 image [224, 224, 3] |
9dtact | vision (9DTact) | uint8 image [224, 224, 3] |
xela | taxel array | float [T, 72] |
tac02 | taxel array | float [T, 66] |
Code & training pipeline: github.com/jxbi1010/HTT · Dataset (with this checkpoint bundled): AllenBi21/HTT-dataset · Project page: jxbi1010.github.io/htt-gh-page
Checkpoint
| File | htt_4sensors_best.pth (~69 MB) |
| Contents | model_state_dict = 4 encoders + shared 9-layer trunk + 4 decoders (17.1 M params) |
| Embedding dim | 192 |
| SHA-256 | 024f4c3a067168197d0a6996bbca5c03e744ed5abd1d35a666dbf78e7ac673f0 |
Slim inference/finetune checkpoint (optimizer / predictor states dropped).
Usage
Use it with the HTT package (contains the architecture, preprocessing, and
examples). Download the weights into checkpoints/:
hf download AllenBi21/HTT htt_4sensors_best.pth --local-dir checkpoints
import numpy as np
from htt import HTT
model = HTT(modality="gsmini") # loads checkpoints/htt_4sensors_best.pth
frame = np.random.randint(0, 256, (224, 224, 3), dtype=np.uint8) # your sensor frame
emb = model(frame) # -> torch.Tensor [1, 192]
Read the raw-input contract before feeding your own data — the model returns garbage on out-of-distribution inputs.
License
Released under CC BY-NC 4.0 (non-commercial). Portions are derived from Meta's V-JEPA / DINOv2 (Apache-2.0 and CC-BY-NC-4.0).