Inverse Dynamics Model for Action-Annotating Screencasts
We present an inverse dynamics model that predicts user input actions from short windows of screen recordings. Given 10 consecutive screenshots, it emits the key presses, mouse clicks, cursor movements and scroll events that are visually implied by the frames.
Please refer to the blog post for details and experiments.
Summary
- Base model:
Qwen/Qwen3-VL-8B-Instruct - Training data:
crowd-castrecordings - Training method: LoRA on language and vision modules, merged after training
- Input: 10 screenshots sampled at 5 FPS
- Output: sparse JSON action list
- Eval:
p-doom/idm-eval-set - Code:
p-doom/inverse-dynamics-model
IDM-eval results
All models are evaluated on our eval set, visibility-filtered to visible + inferable actions. MM R² and MM cos_mean include missed MouseMove frames as zero predictions; MM cov. is MouseMove recall.
| Model | Overall F1 | KeyPress F1 | MouseClick F1 | MouseMove F1 | MouseScroll F1 | MM R² | MM cos_mean | MM cov. |
|---|---|---|---|---|---|---|---|---|
| Ours (8B) | 0.787 | 0.791 | 0.598 | 0.857 | 0.447 | 0.708 | 0.643 | 92% |
| Gemini 3.5 Flash | 0.740 | 0.826 | 0.726 | 0.760 | 0.337 | 0.714 | 0.560 | 64% |
| GPT 5.5 | 0.709 | 0.821 | 0.714 | 0.669 | 0.392 | 0.586 | 0.455 | 52% |
| Kimi K2.6 | 0.540 | 0.711 | 0.444 | 0.381 | 0.326 | 0.420 | 0.177 | 25% |
| Gemma 4 31B | 0.430 | 0.381 | 0.581 | 0.500 | 0.237 | 0.077 | 0.228 | 37% |
| Qwen3-VL 8B | 0.360 | 0.409 | 0.449 | 0.334 | 0.127 | -6.038 | 0.035 | 28% |
Interpretation: the main gap is dense temporal coverage. Off-the-shelf VLMs under-emit MouseMove actions, and the all-GT MouseMove metrics penalize these misses.
Input Format
Provide one chat message with 10 images sampled at 5 FPS. Each image should be preceded by a text label:
Frame F00: <image>
Frame F01: <image>
...
Frame F09: <image>
The frame labels are text anchors in the message, not labels rendered into the image pixels.
Output Format
The model emits only a JSON array:
[
{"frame": "F02", "type": "MouseMove", "details": "120,45"},
{"frame": "F03", "type": "MouseClick", "details": "Left"},
{"frame": "F05", "type": "KeyPress", "details": "Cmd+S"},
{"frame": "F07", "type": "MouseScroll", "details": "-150"}
]
Action types:
KeyPress: key name with modifiers, e.g.Cmd+S,Return,AMouseClick:Left,Right, orMiddleMouseMove: normalizeddx,dy, where1000is a full screen-width or screen-height traversalMouseScroll: normalized signed scroll magnitude
Frame attribution: if an effect first appears between F_K and F_{K+1}, report the action on F_K, the last pre-action frame.
Related Releases
p-doom/AGI-CAST-0.6k: source AGI-CAST screencastsp-doom/AGI-CAST-idm-actions: AGI-CAST action annotations generated with this modelp-doom/idm-eval-set: manually verified IDM evaluation clips
Limitations
- The model was trained on macOS clips and can confuse OS-specific shortcuts such as
CmdvsCtrl. - Labels are inferred from pixels, so actions with no visual evidence can be missed or hallucinated.
- Fine-grained timing, cursor movement magnitude and scroll magnitude can be noisy.