LeVJEPA-VideoMix-Large

A ViT-L/16 video encoder trained with LeV-JEPA, a self-supervised objective combining a multi-crop prediction loss with SIGReg. No labels are used at any point.

• 303.1M parameters, 224px, patch 16, 16 frames, tubelet 1 (one token per frame per patch)
• RoPE position encoding, block-causal attention
• Trained on VideoMix: 1,806,869 clips from Kinetics-710, Something-Something v2, Walking Tours and PE-Video

Usage

import torch
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "galilai-group/LeVJEPA-VideoMix-Large", trust_remote_code=True
).eval()

# (B, C, T, H, W), ImageNet-normalised
video = torch.randn(1, 3, 16, 224, 224)

with torch.no_grad():
    out = model(pixel_values=video)

out.last_hidden_state   # (1, 3137, 1024) -- CLS + 16*14*14 patch tokens
out["pooler_output"]    # (1, 1024)       -- the CLS token

trust_remote_code=True is required: this is a custom architecture (RoPE + block-causal attention) rather than a stock transformers model, so the modeling code ships with the weights.

Preprocessing

Normalise with the ImageNet statistics used in training — mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225] — and resize/crop to 224. Frames are sampled at roughly 7.5 fps in training, so a 16-frame clip covers about two seconds.

For a single image, repeat it along the temporal axis:

image = torch.randn(1, 3, 224, 224)
video = image.unsqueeze(2).repeat(1, 1, 16, 1, 1)

Attention mode

The weights were trained with block-causal attention: bidirectional within a temporal slot, causal across slots, with CLS as a read-only sink that sees the whole clip while no patch attends to it. config.attn_mode defaults to "block_causal" for this reason.

Running these weights under full attention will not raise an error — it will quietly return worse features. Leave attn_mode alone unless you know why you are changing it. An explicit attention mask also disqualifies SDPA's flash kernel, so expect higher memory than a full-attention ViT-L at the same batch size.

Weights

The released tensors are the EMA copy of the encoder (decay=0.9999, update_every=32), which is what we evaluate. There is no separate LR-decay artifact to apply — the cosine leg is already baked into these weights.

Training details

Objective	multi-crop prediction + SIGReg (weight 0.02)
Crops	1 global + 10 local
Token drop	95%, random, applied inside the encoder forward (training only)
Optimizer	AdamW, lr 4e-4 flat then 1-sqrt → 0, weight decay 0.04
Batch	3072 global
Precision	bf16-mixed

Token dropping is a training-time regulariser and is inert under eval(), so the released model returns all 3137 tokens.

Intended use

Frozen feature extraction for video and image understanding — attentive or linear probing, retrieval, and as a backbone for downstream heads. It is a self-supervised encoder with no classification head.