V-JEPA-LM: counterfactual supervision teaches video-language models to read motion
Researchers report that dynamics in a frozen V-JEPA encoder can be located and read by a language model, lifting video question answering above text-aligned baselines.
In plain words
A video encoder can store how scenes change, but language models trained on captions often ignore that information. The authors create training examples where pixel transformations change the correct answer, forcing the model to read the motion itself. The resulting models reportedly score higher on eleven video benchmarks.
What was announced
The arXiv abstract explains the gap: a linear probe reading frozen V-JEPA features identifies speed at 87%, while the paired vision-language model answers at chance on the same items, because captions and questions name only a few attributes and so teach retrieval rather than reading. The authors locate dynamics in a subspace shared across videos for predictive encoders like V-JEPA, but not in frame-wise, text-aligned encoders such as SigLIP.
Their fix is counterfactual supervision: public videos are altered with pixel transformations into answer-balanced variants whose answers depend only on the change. Trained in one stage on 4.53M public samples with 32 visual tokens per frame, the V-JEPA-LM models (1B to 8B, built on Qwen3 and Llama) reportedly lead every size class across eleven benchmarks; the 4B model scores 72.1 versus 61.1 for the best open 4B and 64.8 for the best open 8B. A controlled study attributes 4.5 average points, and 38 on speed questions, to the supervision at matched budget. Code and models are linked on GitHub.
What it does not show
- Evidence is limited to the paper's abstract; benchmark scores, the encoder-subspace finding and the supervision gains are author-reported and not independently confirmed.
- The abstract does not state which eleven benchmarks were used, how the models compare on individual datasets, or any availability details beyond a code link.
Sources
Source-support review completed 17 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 17 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.