AffectFlow-DINO
Uncertainty-aware multi-task facial affect estimation for the 11th ABAW Multi-Task Learning challenge: DINOv3 backbone + deterministic VA/expression/AU heads + a conditional rectified-flow head over the joint 22-dimensional affect vector.
Code, training scripts, and the full 28-experiment ablation study: github.com/Bekhouche/AffectFlow-DINO.
Paper: AffectFlow-DINO: Uncertainty-Aware Multi-Task Affect Estimation via Conditional Rectified Flow — Salah Eddine Bekhouche, Abdellah Zakaria Sellam, Fadi Dornaika, Abdenour Hadid. arXiv:2607.13250, 2026.
Quickstart
pip install torch torchvision transformers huggingface_hub pillow
git clone https://github.com/Bekhouche/AffectFlow-DINO.git && cd AffectFlow-DINO
python inference.py --model finetune-flow-retune-b10 --image face.jpg
Models in this repo
Each subfolder is one checkpoint: model.pt (weights + architecture config) plus, where
applicable, au_thresholds.json and/or expr_weights.json post-hoc calibration files that
inference.py applies automatically.
| Folder | Backbone | Best decode | P_MTL (calibrated) | P_VA | P_EXPR | P_AU |
|---|---|---|---|---|---|---|
finetune-flow-retune-b10 | ViT-S/16, fine-tuned | Det | 1.177 | 0.325 | 0.350 | 0.502 |
vitb-finetune | ViT-B/16, fine-tuned | Det | 1.116 | 0.354 | 0.255 | 0.507 |
finetune-flow-retune-b05 | ViT-S/16, fine-tuned | Det / Flow | 1.062 / 0.956 | 0.291 | 0.303 | 0.469 |
finetune | ViT-S/16, fine-tuned | Det | 1.101 | 0.318 | 0.285 | 0.497 |
au-posw-cw | ViT-S/16, frozen | Det | 0.890 (no cal.) | 0.211 | 0.240 | 0.439 |
psp | ViT-S/16, frozen (patch soft-pool) | Det | 0.859 (no cal.) | 0.235 | 0.236 | 0.389 |
affectflow-base | ViT-S/16, frozen | Flow | 0.888 | 0.237 | 0.210 | 0.441 |
det-baseline | ViT-S/16, frozen | Det | 0.793 (no cal.) | 0.199 | 0.216 | 0.378 |
P_MTL = P_VA + P_EXPR + P_AU is the official ABAW MTL composite metric (P_VA = mean CCC of
valence/arousal, P_EXPR = expression macro-F1, P_AU = mean AU F1), evaluated on the s-Aff-Wild2
validation split. Official challenge baseline: P_MTL = 0.450.
These 8 checkpoints are a curated subset of the ~30 ablation runs behind the paper — one per architecturally distinct configuration. The full ablation table, including sweeps that reused one of these architectures with different loss weights/regularization, is in the GitHub repo's EXPERIMENTS.md.
Input / output
- Input: a cropped, roughly frontal face image, resized to 224x224 and ImageNet-normalized
(handled automatically by
inference.py). - Output: valence, arousal in
[-1, 1]; one of 8 expressions (Neutral, Anger, Disgust, Fear, Happiness, Sadness, Surprise, Other); a subset of 12 active Action Units (AU1, AU2, AU4, AU6, AU7, AU10, AU12, AU15, AU23, AU24, AU25, AU26). - Two decode modes:
deterministic(task heads directly) orflow(average of N sampled rectified-flow trajectories — enables uncertainty estimation via trajectory spread).
Limitations
Trained and validated only on s-Aff-Wild2 (in-the-wild but frame-level, no temporal context). Expression prediction remains the weakest task (P_EXPR <= 0.35): Fear and Sadness are rare classes that only partially recover under calibration. Not evaluated on demographic subgroups; treat outputs as research signals, not clinical or high-stakes decisions.
Citation
@article{bekhouche2026affectflowdino,
title = {AffectFlow-DINO: Uncertainty-Aware Multi-Task Affect Estimation via Conditional Rectified Flow},
author = {Bekhouche, Salah Eddine and Sellam, Abdellah Zakaria and Dornaika, Fadi and Hadid, Abdenour},
journal = {arXiv preprint arXiv:2607.13250},
year = {2026}
}
License
MIT.