TCAM: Track and Caption Any Motion
Query-free motion discovery and description in videos. Code: ostadabbas/TCAM-Track-and-Caption-Any-Motion
Same drawer, opposite motions. TCAM tracks what moves and tells them apart: opening (orange, out) vs. closing (blue, in).
Give TCAM a video and it tells you what moves, when, and where: timestamped captions for each motion event, each grounded to the point tracks of the subject performing it. No query needed.
Quick start
git lfs install
git clone https://huggingface.co/bishoygaloaa/tcam && cd tcam
pip install torch==2.3.1 torchvision==0.18.1 --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt
python infer.py --video my_clip.mp4 --vis
=== my_clip.mp4
[ 0.00s - 4.20s] A boy is skating on a zebra crossing.
[ 4.68s - 7.52s] He is skating on a zebra crossing.
-> tcam_out/my_clip.json
--vis also writes tcam_out/my_clip_vis.mp4, which overlays each event's grounded
points on the video. The TCAM weights download automatically, and so does CoTracker3,
from Meta's official facebook/cotracker3 repo.
Useful flags: --chunk_sec 10 for long videos, --start_sec/--end_sec for a segment,
--sample --seed 1 for more varied event windows, --model <local dir> to run offline.
Python
from tcam import TCAMPipeline
pipe = TCAMPipeline.from_pretrained("bishoygaloaa/tcam")
res = pipe("my_clip.mp4")
for e in res.events:
print(e.start_sec, e.end_sec, e.caption)
xy = res.tracks[:, e.top_tracks(40)] # grounded points, (16 frames, 40, 2), normalised
mask = res.masks(event=0) # (16, 224, 224) subject mask
res.save_json("my_clip.json")
Fine-tuning
Describe your videos in a train.json (times in seconds; box is optional and
normalised to [0, 1]. Add it to also train grounding):
[{"video": "clips/drawer.mp4",
"events": [{"start": 0.0, "end": 3.5, "caption": "a hand opens the drawer", "box": [0.2, 0.3, 0.8, 0.7]},
{"start": 3.5, "end": 7.0, "caption": "the hand closes the drawer"}]}]
python finetune.py --data train.json --out_dir my_tcam --steps 1000
python infer.py --model my_tcam --video new_clip.mp4
This uses the same objective as the original training. Keep --lr low (default 2e-5).
Notes
- TCAM reads 16 frames per call. Use
--chunk_secfor anything longer than a short clip. - Grounding selects the subject's tracks; masks are drawn from those tracks.
- Tested on a CUDA GPU with Python 3.10. Keep the pinned versions in
requirements.txt.
License
Weights and code: CC BY-NC 4.0 (non-commercial). FLAN-T5, CLIP and CoTracker3 keep their own licenses.
Citation
@inproceedings{tcam2025,
title={Track and Caption Any Motion: Query-Free Motion Discovery and Description in Videos},
author={Bishoy Galoaa and Sarah Ostadabbas},
year={2026}
}