LeVJEPA-VideoMix-Large
A ViT-L/16 video encoder trained with LeV-JEPA, a self-supervised objective combining a multi-crop prediction loss with SIGReg. No labels are used at any point.
- 303.1M parameters, 224px, patch 16, 16 frames, tubelet 1 (one token per frame per patch)
- RoPE position encoding, block-causal attention
- Trained on VideoMix: 1,806,869 clips from Kinetics-710, Something-Something v2, Walking Tours and PE-Video
Usage
import torch
from transformers import AutoModel
model = AutoModel.from_pretrained(
"galilai-group/LeVJEPA-VideoMix-Large", trust_remote_code=True
).eval()
# (B, C, T, H, W), ImageNet-normalised
video = torch.randn(1, 3, 16, 224, 224)
with torch.no_grad():
out = model(pixel_values=video)
out.last_hidden_state # (1, 3137, 1024) -- CLS + 16*14*14 patch tokens
out["pooler_output"] # (1, 1024) -- the CLS token
trust_remote_code=True is required: this is a custom architecture (RoPE +
block-causal attention) rather than a stock transformers model, so the modeling
code ships with the weights.
Preprocessing
Normalise with the ImageNet statistics used in training —
mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225] — and resize/crop to 224.
Frames are sampled at roughly 7.5 fps in training, so a 16-frame clip covers about
two seconds.
For a single image, repeat it along the temporal axis:
image = torch.randn(1, 3, 224, 224)
video = image.unsqueeze(2).repeat(1, 1, 16, 1, 1)
Attention mode
The weights were trained with block-causal attention: bidirectional within a
temporal slot, causal across slots, with CLS as a read-only sink that sees the whole
clip while no patch attends to it. config.attn_mode defaults to "block_causal"
for this reason.
Running these weights under full attention will not raise an error — it will quietly return worse features. Leave
attn_modealone unless you know why you are changing it. An explicit attention mask also disqualifies SDPA's flash kernel, so expect higher memory than a full-attention ViT-L at the same batch size.
Weights
The released tensors are the EMA copy of the encoder (decay=0.9999,
update_every=32), which is what we evaluate. There is no separate LR-decay artifact
to apply — the cosine leg is already baked into these weights.
Training details
| Objective | multi-crop prediction + SIGReg (weight 0.02) |
| Crops | 1 global + 10 local |
| Token drop | 95%, random, applied inside the encoder forward (training only) |
| Optimizer | AdamW, lr 4e-4 flat then 1-sqrt → 0, weight decay 0.04 |
| Batch | 3072 global |
| Precision | bf16-mixed |
Token dropping is a training-time regulariser and is inert under eval(), so the
released model returns all 3137 tokens.
Intended use
Frozen feature extraction for video and image understanding — attentive or linear probing, retrieval, and as a backbone for downstream heads. It is a self-supervised encoder with no classification head.