Llama-V-JEPA-LM-Qwen3-30B-A3B

Llama-V-JEPA-LM is a series of video language models by Kim et al. (paper arXiv 2611.17831); previous issues introduced smaller sizes such as 1.7B. This largest version connects the frozen Meta V-JEPA 2.1 ViT-G video encoder to a fully fine-tuned Qwen3-30B-A3B via a "spectral bridge," with a total of 32.39B parameters. Each video segment samples 32 frames and converts them into 1,024 visual tokens, a number independent of input resolution. Training data includes public video question-answering sets, as well as "counterfactual" samples with deterministic answers constructed using transformations such as reversal, mirroring, and speed changes, allowing the model to learn how visual content changes. The authors report an average score of 69.49 across 12 items in 11 video benchmarks, higher than MiniCPM-V-4.7-35B-A3B's 66.41 and Qwen3-VL-30B-A3B's 63.33.

Llama-V-JEPA-LM is a series of video language models by Kim et al. (paper arXiv 2611.17831); previous issues introduced smaller sizes such as 1.7B. This largest version connects the frozen Meta V-JEPA 2.1 ViT-G video encoder to a fully fine-tuned Qwen3-30B-A3B via a "spectral bridge," with a total of 32.39B parameters. Each video segment samples 32 frames and converts them into 1,024 visual tokens, a number independent of input resolution. Training data includes public video question-answering sets, as well as "counterfactual" samples with deterministic answers constructed using transformations such as reversal, mirroring, and speed changes, allowing the model to learn how visual content changes. The authors report an average score of 69.49 across 12 items in 11 video benchmarks, higher than MiniCPM-V-4.7-35B-A3B's 66.41 and Qwen3-VL-30B-A3B's 63.33.

Why it matters The authors claim the average score is higher than that of Qwen3-VL-30B-A3B, which shares the same base model, with the largest gap observed on custom holdout tests involving reversal, speed changes, and mirroring.

Hugging Face likes 12 · Hugging Face upvotes 12 · Hugging Face downloads 3

License: Commercial use allowed · ComfyUI: Not available · VRAM: ≈78 GB estimated

Reasons

  • Includes paper, technical report, and Hugging Face online demo
  • The evaluation table also lists areas where it underperforms, such as video scores on MotionBench and Vinoground

Usage

  • Install dependencies such as transformers 5 or higher, safetensors, and eva-decord. Load the model using AutoModel.from_pretrained("kkt94/Llama-V-JEPA-LM-Qwen3-30B-A3B", trust_remote_code=True, dtype=torch.bfloat16). Use model.chat("my_video.mp4", "question") for free-form answers and model.choose for multiple-choice questions. For long videos, use start_sec / end_sec to select segments.

Risks

  • Weights are approximately 65 GB; the authors recommend a GPU with 80 GB of VRAM (the authors measured a peak of 61.0 GiB for a single segment). Requires trust_remote_code.
  • Code and weights are labeled Apache-2.0, but multiple training datasets prohibit commercial use. The authors state that the weights are released for research and non-commercial use, and are subject to the Llama 3.2 usage policy.
  • Trained only on English; OCR, single images, and long videos are not its targets.

Open

抱脸 world-model 标签

Explore the latest AI Radar updates