V-JEPA2 Enhanced Encoder for Video-Evoked BOLD Responses

This repository contains an enhanced V-JEPA2-large fMRI encoder trained to predict video-evoked BOLD responses on the fsaverage surface. It keeps the same V-JEPA2 backbone family as the basic encoder and improves the readout with 24 layer features, PLSTorch rank-16 decoders, and 32-member bootstrap bagging with member-wise select-then-average layer selection.

The default AutoModel variant is enhanced_all, trained on all available BoldMoments/Lahner2024 and McMahon2023 datasets.

The dashed reference is the all-vertex human ceiling after inverse Spearman-Brown correction, square root transform, and Spearman-Brown correction.

Input/Output Contract

• Input: one 3-second RGB video clip sampled at roughly 12 fps (about 36 frames -- the setting the encoder was trained on), represented as a float tensor shaped [B, T, C, H, W] (so T is about 36) with values in [0, 1].
• Output: one vector of predicted z-scored fMRI beta responses per video, shaped [B, 20484].
• Decoder-only input: precomputed pooled V-JEPA2 layer features can be passed to forward_features(features); model.decoders exposes one entry per feature layer, each with mean/std shaped [1, feature_dim].

The video-input path resizes frames to 224 x 224 and applies ImageNet normalization before V-JEPA2 feature extraction.

Loading

import torch
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "epfl-neuroai/vjepa2-encoder-enhanced",
    trust_remote_code=True,
)
model.eval()

# Replace this with a preprocessed 3-second video clip at ~12 fps (about 36 frames),
# which is the setting the encoder was trained on.
# Shape: [batch, frames, channels, height, width].
video = torch.zeros(1, 36, 3, 224, 224)

with torch.no_grad():
    prediction = model.predict_fmri(video)

print(prediction.shape)  # [1, 20484]

For decoder-only debugging, the model can also run from precomputed V-JEPA2 layer features:

model = AutoModel.from_pretrained(
    "epfl-neuroai/vjepa2-encoder-enhanced",
    trust_remote_code=True,
    load_vjepa=False,
)

features = [
    torch.zeros(1, decoder.mean.shape[1])
    for decoder in model.decoders
]

with torch.no_grad():
    prediction = model.forward_features(features)

Variants

• enhanced_all (default): enhanced decoder trained on all BMD/Lahner + McMahon data.
• basic_all: six-layer Ridge baseline trained on all BMD/Lahner + McMahon data.
• enhanced_joint_train_to_joint_val: enhanced decoder trained on joint train split and evaluated on joint validation.
• basic_joint_train_to_joint_val: matched six-layer Ridge joint validation baseline.
• enhanced_bmd_to_mcmahon, basic_bmd_to_mcmahon: train on BMD/Lahner, transfer to McMahon.
• enhanced_mcmahon_to_bmd, basic_mcmahon_to_bmd: train on McMahon, transfer to BMD/Lahner.

Pass variant=... to from_pretrained to load a non-default checkpoint.

Held-Out Metrics

Data

This checkpoint was trained using data from the BOLD Moments Dataset (BMD/Lahner) and the McMahon social interaction video fMRI dataset. This repository does not include the underlying fMRI datasets or stimulus videos.

Files

• enhanced_all.pth: default enhanced all-data decoder checkpoint.
• basic_*.pth, enhanced_*.pth: comparison and evaluation decoder checkpoints.
• vitl.pt: local V-JEPA2-large backbone weights.
• config.json, configuration_vjepa2_fmri_encoder.py, modeling_vjepa2_fmri_encoder.py: custom Transformers files for AutoModel loading.
• metrics.json: held-out metrics used by the model card.
• assets/comparison_metrics.png: joint validation comparison plot.

Backbone Source

The loader uses the V-JEPA2 Torch Hub architecture with pretrained=False, then loads the local vitl.pt weights directly. This avoids relying on a moving external checkpoint URL while preserving compatibility with the decoder feature hooks.

Citations

If you use this checkpoint, please cite the V-JEPA/V-JEPA 2 backbone papers and source datasets:

@article{tang2025diverse,
  title={Diverse perceptual representations across visual pathways emerge from a single objective},
  author={Tang, Yingtian and Gokce, Abdulkadir and Al-Karkari, Khaled Jedoui and Yamins, Daniel and Schrimpf, Martin},
  journal={bioRxiv},
  pages={2025--07},
  year={2025},
  publisher={Cold Spring Harbor Laboratory}
}

@article{bardes2024revisiting,
  title={Revisiting Feature Prediction for Learning Visual Representations from Video},
  author={Bardes, Adrien and Garrido, Quentin and Ponce, Jean and Chen, Xinlei and Rabbat, Michael and LeCun, Yann and Assran, Mahmoud and Ballas, Nicolas},
  journal={arXiv preprint arXiv:2404.08471},
  year={2024}
}

@article{assran2025vjepa2,
  title={V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning},
  author={Assran, Mido and Bardes, Adrien and Fan, David and Garrido, Quentin and Howes, Russell and Komeili, Mojtaba and Muckley, Matthew and Rizvi, Ammar and Roberts, Claire and Sinha, Koustuv and others},
  journal={arXiv preprint arXiv:2506.09985},
  year={2025}
}

@article{lahner2024modeling,
  title={Modeling short visual events through the BOLD moments video fMRI dataset and metadata},
  author={Lahner, Benjamin and Dwivedi, Kshitij and Iamshchinina, Polina and Graumann, Monika and Lascelles, Alex and Roig, Gemma and Gifford, Alessandro Thomas and Pan, Bowen and Jin, SouYoung and Ratan Murty, N Apurva and others},
  journal={Nature communications},
  volume={15},
  number={1},
  pages={6241},
  year={2024},
  publisher={Nature Publishing Group UK London}
}

@article{mcmahon2023hierarchical,
  title={Hierarchical organization of social action features along the lateral visual pathway},
  author={McMahon, Emalie and Bonner, Michael F and Isik, Leyla},
  journal={Current Biology},
  volume={33},
  number={23},
  pages={5035--5047},
  year={2023},
  publisher={Elsevier}
}