Open-VLJEPA — MSRVTT in-domain checkpoint (R@1 = 23.9)
Trained checkpoint for Open-VLJEPA — an open re-implementation of VL-JEPA (Chen et al., 2026, Meta).
Note — this is a resource-poor re-implementation. Training was limited to a single workstation (8 × RTX 4090, 192 GB VRAM total) over a few days, versus the paper's 192 × H200 for 4 weeks on ~3.3 B samples. Numbers below reflect that gap.
My work runs on caffeine ☕. If this checkpoint is useful to you, a small donation helps keep the lights on and more open re-implementations coming.
Results (MSRVTT test, 500-video pool)
This checkpoint is in-domain MSRVTT Stage B (50 epochs), initialized from a CC3M image-text Stage A.
| Metric | T2V | V2T |
|---|---|---|
| R@1 | 23.90 | — |
| R@5 | 51.44 | — |
| R@10 | 65.05 | — |
See eval_history.json for the full 50-epoch trajectory.
Architecture
- X-Encoder:
facebook/vjepa2-vitl-fpc64-256(frozen, weights NOT in this ckpt — re-fetched from HF at load time) - Predictor: last 8 layers of
meta-llama/Llama-3.2-1B+ projection, bi-directional attention - Y-Encoder:
google/embeddinggemma-300m+ projection - Loss: bi-directional InfoNCE in a shared 1536-D space
Only the trainable parts (predictor + y_encoder + projections, ~800 M params) are saved in best.pt. The frozen V-JEPA 2 encoder is loaded directly from facebook/vjepa2-vitl-fpc64-256 at inference time.
Usage
git clone https://github.com/dion-jy/open-vljepa
cd open-vljepa
pip install torch transformers webdataset decord pyyaml huggingface-hub
# Download ckpt
huggingface-cli download cun-bjy/open-vljepa best.pt --local-dir checkpoints_msrvtt
# Demo
python scripts/demo_gif.py --ckpt checkpoints_msrvtt/best.pt
# Eval
python scripts/eval.py --ckpt checkpoints_msrvtt/best.pt --n_videos 500
Training
- Stage A — image-text on CC3M (~1 M pairs, 1 frame per visual input), 10 ep
- Stage B (this ckpt) — MSRVTT, 50 ep, init from Stage A best
- 8 × RTX 4090, bs = 64/gpu (effective contrastive pool 512 via all-gather), LR 1e-4, bf16 + gradient checkpointing
Configs are in the GitHub repo (configs/stage_a.yaml, configs/stage_b_retrain.yaml).
License & Attribution
Built with Llama (meta-llama/Llama-3.2-1B, Llama 3.2 Community License) and Gemma (google/embeddinggemma-300m, Gemma License). V-JEPA 2 (facebook/vjepa2-vitl-fpc64-256, MIT) is used in frozen form.
By using this checkpoint you agree to:
- The Llama 3.2 Acceptable Use Policy
- The Gemma Acceptable Use Policy
Citation
@article{chen2026vljepa,
title = {VL-JEPA: Joint Embedding Predictive Architecture for Vision-Language},
author = {Chen et al.},
journal = {arXiv preprint arXiv:2512.10942},
year = {2026}
}
@misc{openvljepa2026,
title = {Open-VLJEPA: a small-scale open re-implementation of VL-JEPA},
author = {Baek, Junyeob},
year = {2026},
howpublished = {\url{https://github.com/dion-jy/open-vljepa}},
note = {GitHub repository}
}