DensFiLM — Density-Conditioned Video Saliency for Crowd Scenes
DensFiLM is a Video Swin Transformer (Swin3D-S) augmented with a FiLM layer at the encoder bottleneck. A 64-dimensional learnable crowd-density embedding is projected to per-channel scale–shift parameters (γ, β) ∈ ℝ⁷⁶⁸ that recalibrate the most semantically abstract spatiotemporal feature map before decoding — at a cost of only ~100K additional parameters (<0.2% of backbone weights).
The motivation: as crowd density rises, human fixation strategy shifts categorically — from tracking individual pedestrians, to reading collective motion structure (flow lanes, merge boundaries), to localising scene-level landmarks against density gradients. Existing video saliency models treat all densities uniformly.
- Paper: arXiv:2607.25465
- Code: github.com/aniskhan25/crowdfix-saliency
- Author: Anis Ur Rahman, CSC – IT Center for Science Ltd., Espoo, Finland
Results on CrowdFix
5,356-clip CrowdFix test set. DensFiLM rows are mean ± std over 4 seeds; baselines are as published in the 2019 CrowdFix benchmark paper.
| Model | AUC-J ↑ | AUC-B ↑ | NSS ↑ | CC ↑ | KL ↓ | SIM ↑ |
|---|---|---|---|---|---|---|
| SAM | 0.777 | – | 0.894 | 0.325 | – | – |
| DeepVS | 0.788 | – | 0.985 | 0.401 | – | – |
| ACL-Net (2019) | 0.817 | – | 1.250 | 0.450 | – | – |
| Swin3D-S, unconditioned | 0.806 | 0.810 | 1.313 | 0.476 | 1.133 | 0.424 |
| DensFiLM (oracle label) | 0.820 ±0.001 | 0.825 ±0.001 | 1.434 ±0.007 | 0.517 ±0.002 | 1.068 ±0.004 | 0.449 ±0.003 |
| DensFiLM (predicted label) | 0.821 | 0.825 | 1.438 | 0.518 | 1.067 | 0.451 |
This surpasses ACL-Net — the sole published CrowdFix benchmark since 2019 — by +14.7% NSS and +14.9% CC.
Notably, replacing the ground-truth density label with the model's own density head prediction gives identical performance (ΔNSS ≤ 0.001), so the conditioning signal is self-derived at inference and no density annotation is needed.
Usage
import torch
from models.densfilm_hub import DensFiLM
model = DensFiLM.from_pretrained("aniskhan25/densfilm-crowdfix").eval()
# (B, 3, T, H, W) — 8 frames at 224x384, ImageNet-normalised
clip = torch.randn(1, 3, 8, 224, 384)
with torch.no_grad():
saliency, density_logits = model(clip) # density self-derived
# saliency: (1, 1, 224, 384) in [0, 1]
# density_logits: (1, 3) over SP / DF / DC
print(saliency.shape, density_logits.argmax(-1))
To condition on a known density category instead:
density = torch.tensor([2]) # 0=SP, 1=DF, 2=DC
saliency, _ = model(clip, density)
Preprocessing
Frames must be resized to 224×384 and ImageNet-normalised
(mean [0.485, 0.456, 0.406], std [0.229, 0.224, 0.225]), stacked into
clips of 8 consecutive frames as (B, 3, T, H, W).
import torch
import torchvision.transforms.functional as TF
MEAN = torch.tensor([0.485, 0.456, 0.406]).view(3, 1, 1)
STD = torch.tensor([0.229, 0.224, 0.225]).view(3, 1, 1)
def preprocess(frames): # list of 8 PIL images
t = torch.stack([TF.to_tensor(TF.resize(f, [224, 384], antialias=True))
for f in frames]) # (T, 3, H, W)
t = (t - MEAN) / STD
return t.permute(1, 0, 2, 3).unsqueeze(0) # (1, 3, T, H, W)
Density categories
| Label | Code | Meaning |
|---|---|---|
| 0 | SP | Sparse — individual pedestrian trajectories dominate fixation |
| 1 | DF | Dense Flow — collective motion structure, flow lanes, merge boundaries |
| 2 | DC | Dense Congested — scene-level semantic landmarks against density gradients |
Training
Trained on CrowdFix (3,189 training clips) from a Kinetics-400-pretrained Swin3D-S backbone; FiLM layers, density embedding and decoder initialise fresh.
| Setting | Value |
|---|---|
| Epochs | 150, early stopping patience 20 |
| Optimiser | AdamW, weight decay 1e-4, cosine annealing |
| LR | 1e-4 (backbone 0.1×), 20 frozen-backbone warmup epochs |
| Batch size | 2 per GPU × 8 GCDs |
| Clip | 8 frames at 224×384 |
| Hardware | 8× AMD MI250X GCDs on LUMI |
Reported figures are mean ± std over 4 seeds. This checkpoint is a single seed, so its individual metrics will sit within the quoted std rather than exactly on the mean.
Limitations
- Trained and evaluated only on CrowdFix; generalisation to non-crowd video saliency is untested. Domain shift to other crowd datasets is unmeasured.
- 3,189 training clips place the model firmly in the regularisation-dominated regime. The paper's ablations show that adding capacity (3D temporal decoder, learned social-force prior) collapses NSS to roughly the unconditioned baseline (~1.313).
- Explicit optical flow is redundant: a RAFT dual-stream variant lands within seed variance (NSS 1.437 vs 1.434 ± 0.007), as Swin3D-S's shifted-window attention already captures the motion signal.
- Debiased evaluation (subtracting mean training fixation density to remove shared centre-prior alignment) drops DensFiLM to AUC-J 0.765 / NSS 1.188. The unconditioned backbone loses 45% of its NSS while DensFiLM loses only 17% — density conditioning substitutes genuine structure for centre prior, but a centre-prior component remains.
- Fixed input geometry: 8 frames at 224×384. Other resolutions are untested.
Citation
@article{rahman2026densfilm,
title = {Density-Conditioned Video Saliency for Crowd Scenes via
Feature-wise Linear Modulation},
author = {Rahman, Anis Ur},
journal = {arXiv preprint arXiv:2607.25465},
year = {2026}
}