PG-XFormer
Pose-guided cross-modal exercise recognition with a safety-gated feedback layer.
PG-XFormer fuses a CTR-GCN skeleton stream and a VideoMAE video stream with symmetric
cross-attention to recognise physiotherapy exercises from a short clip and its 2D pose.
It is the recognition backbone of the paper "PG-XFormer: Pose-Guided Cross-Modal Exercise
Monitoring with Agentic Feedback". This repository holds the trained weights for every
configuration and seed reported in the paper; code, split definitions, per-clip predictions,
and the feedback audit are in
PG-XFormer-artifacts.
All PG-XFormer resources are grouped in the PG-XFormer collection.
Model details
| Inputs | 16 RGB frames at 224 × 224 (values in [0, 1]) and 16 × 17 COCO-17 2D keypoints, pelvis-centred and divided by the shoulder-to-hip length |
| Skeleton stream | CTR-GCN, 7 blocks (64 → 256 channels), temporal kernel 9 — 1.9M parameters, ~1 GFLOP/clip |
| Video stream | VideoMAE-base (Kinetics-pretrained), lower 8 of 12 blocks frozen — 86M parameters, ~361 GFLOPs/clip |
| Fusion | 4 layers of symmetric skeleton↔video cross-attention (8 heads, width 512), then concatenation, LayerNorm, mean-pool |
| Output | 10 InfiniteRep exercises: arm raise, bicycle crunch, bird-dog, curl, fly, leg raise, overhead press, push-up, squat, superman |
| Size / cost | 114.7M parameters, 416 GFLOPs per clip; ~85 ms per clip on a 6 GB consumer GPU (batch 1, fp16) |
Checkpoints
model.safetensors is the default checkpoint (cross-attention, seed 0; fixed in advance as
the initialisation of the clinical experiments, not chosen by test score).
checkpoints/synthetic/<run>.safetensors holds the other runs, named
<fusion>_<objective>_s<seed> (ce: cross-entropy; align: + InfoNCE, λ = 0.1).
Held-out test results (70 clips, seven per exercise):
| Run | Macro-F1 (%) | ECE |
|---|---|---|
xattn_ce_s0 (default, model.safetensors) | 95.6 | 0.036 |
xattn_ce_s2 | 100.0 | 0.003 |
xattn_align_s2 | 94.2 | 0.046 |
concat_ce_s2 | 97.2 | 0.027 |
concat_align_s2 | 97.0 | 0.026 |
bottleneck_ce_s1 | 97.1 | 0.025 |
bottleneck_ce_s2 | 98.6 | 0.018 |
video_ce_s1 / video_ce_s2 (video only) | 94.1 / 98.6 | 0.046 / 0.017 |
skeleton_ce_s1 / skeleton_ce_s2 (skeleton only) | 66.1 / 63.5 | 0.109 / 0.146 |
Runs without a listed checkpoint (e.g. xattn_ce_s1) were trained before checkpoint saving
was enabled; their per-clip predictions are in the artifacts repository.
Usage
pip install "torch>=2.4" "transformers>=5" safetensors huggingface_hub numpy
from huggingface_hub import hf_hub_download
import importlib.util, sys
spec = importlib.util.spec_from_file_location("pgx", hf_hub_download("FatimahEmadEldin/PG-XFormer", "pgxformer_load.py"))
pgx = importlib.util.module_from_spec(spec); spec.loader.exec_module(pgx)
model, E = pgx.load_pgxformer("xattn_ce_s0") # or e.g. "concat_ce_s2"
logits = model(video, keypoints) # video: (B,16,3,224,224); keypoints: (B,16,17,2)
keypoints must be normalised with E.normalise_keypoints. The checkpoints use the
transformers>=5 VideoMAE parameter naming.
Training
- Data: synthetic InfiniteRep renders (archive), 700 clips of ten exercises; instance-disjoint, class-stratified split of 560 / 70 / 70 clips (split files in the artifacts repository). Keypoints were estimated from the RGB frames with an off-the-shelf 2D pose estimator.
- Procedure: AdamW with cosine schedule, 20 epochs, effective batch 8 (4 × 2 accumulation), fp16 autocast; fusion learning rate 1e-4 (concatenation 3e-4); checkpoint selected on the development split; three seeds per configuration.
Evaluation (mean ± s.d. over three seeds)
| Configuration | Macro-F1 (%) | ECE |
|---|---|---|
| Skeleton only (CTR-GCN) | 65.9 ± 2.4 | 0.131 ± 0.020 |
| Video only (VideoMAE) | 96.6 ± 2.3 | 0.031 ± 0.015 |
| Concatenation fusion | 97.6 ± 0.8 | 0.025 ± 0.009 |
| Bottleneck fusion | 96.2 ± 2.9 | 0.027 ± 0.010 |
| Cross-attention fusion | 98.5 ± 2.6 | 0.017 ± 0.017 |
Fusion is significantly better than the skeleton stream (McNemar p < 1e-18). Differences between the fusion variants and video alone are not statistically significant on the 70-clip test set; cross-attention is significantly better calibrated than bottleneck fusion. Adding the InfoNCE term did not help (98.5 → 96.2%).
Clinical transfer (KIMORE). With a leakage-free five-fold protocol, synthetic pretraining gave no benefit over KIMORE-only training after fine-tuning (93.1 vs. 93.7% macro-F1), although frozen synthetic features reached 72.4% under a linear probe (chance 20%). KIMORE-adapted weights are not released because KIMORE is distributed by its authors on request.
Feedback layer
In the paper the recogniser is paired with a kinematic error report and a language-model cue generator enclosed by a rejection gate (clean and critical reports answered deterministically) and an output verifier (cues naming unreported body regions are replaced). The gate removed unrequested advice on clean repetitions (73.4% → 0% for Qwen3-235B) and made Qwen3-0.6B / 1.7B equally safe. Its code and every generated cue are in the artifacts repository.
Intended use and limitations
- Research on exercise recognition and safe automated feedback. Not a medical device and not validated for clinical decision-making; no clinician has rated the feedback.
- Trained only on synthetic renders of ten exercises; accuracy on real footage, other exercises, or other camera set-ups is unknown. The 70-clip test set limits statistical power.
- Keypoint quality matters: the skeleton stream is weak on estimated synthetic keypoints.
Licence
CC BY-NC 4.0, inherited from the VideoMAE-base weights used for initialisation. InfiniteRep is a synthetic dataset by Infinity AI (research use with attribution).
Citation
Citation to be added on publication. Please also cite InfiniteRep, KIMORE (Capecci et al., IEEE TNSRE 2019), CTR-GCN (Chen et al., ICCV 2021), and VideoMAE (Tong et al., NeurIPS 2022).