WeMM-Video: Tencent's Open-Source Video Understanding Model That Reads Audio and Visuals Together

The MoE version in Tencent's WeMM-Video series (dense models of 2B, 4B, and 9B, plus the 35B-A3B MoE), with a total of 35B parameters and approximately 3B active parameters. It accepts text, images, audio, and video as input and outputs text in Chinese or English. Visual and audio encoders convert media into tokens for the language model; video frames include timestamps, enabling the model to describe events, transcribe speech, and reason about what it sees and hears by combining the original audio track. It supports two modes: direct answer and thinking. The entire series is pre-trained on approximately 1.2 trillion multimodal tokens. Post-training employs multi-teacher OPD, MrDPO for repetition resistance, and specialized reinforcement learning. Technical report: arXiv 2611.26409.

Original title: tencent/WeMM-Video-35BA3B

The MoE version in Tencent's WeMM-Video series (dense models of 2B, 4B, and 9B, plus the 35B-A3B MoE), with a total of 35B parameters and approximately 3B active parameters. It accepts text, images, audio, and video as input and outputs text in Chinese or English. Visual and audio encoders convert media into tokens for the language model; video frames include timestamps, enabling the model to describe events, transcribe speech, and reason about what it sees and hears by combining the original audio track. It supports two modes: direct answer and thinking. The entire series is pre-trained on approximately 1.2 trillion multimodal tokens. Post-training employs multi-teacher OPD, MrDPO for repetition resistance, and specialized reinforcement learning. Technical report: arXiv 2611.26409.

Why it matters Most open-source video models only analyze visuals; this model reads the original audio and timestamped frames together, performing description, transcription, and reasoning in a single pass, with only about three billion parameters active per token.

Rising score 27 · Likes 27 · Downloads 239

Reasons

  • Officially claimed to achieve state-of-the-art open-source performance in full-modality video understanding, surpassing closed-source models such as Qwen3.5-Omni-Flash and Seed2.0-Mini-0428, and narrowing the gap with Gemini-3.0-Flash; the 35B-A3B and 9B models lead in general audio understanding (results presented in charts)
  • Officially released four quantized versions: FP8 Block128, AWQ W4A16, GGUF Q4_K_M, and NVFP4 W4A16; the same quantized versions are also available for the 2B / 4B / 9B models in the series
  • Apache 2.0 license; third-party components are subject to their respective original licenses

Usage

  • Load tencent/WeMM-Video-35BA3B using transformers (trust_remote_code=True); run python examples/transformers\_video.py --model tencent/WeMM-Video-35BA3B --video clip.mp4 --prompt "…" from the root directory of the WeMM-Video repository to process visuals and original audio simultaneously
  • For deployment, first install the official vLLM / SGLang patches; the example command uses two-GPU tensor parallelism, a context length of 8192, and exposes an OpenAI-compatible interface. Set video\_url.use\_audio to true to include the audio track, and chat\_template\_kwargs.enable\_thinking to enable thinking mode
  • If VRAM is insufficient, switch to the 9B, 4B, or 2B models in the same series, or use a quantized version

Risks

  • BF16 weights require approximately 67.39 GiB; the official example uses two GPUs. vLLM / SGLang must be patched for WeMM, and the repository's bundled code must be run with trust\_remote\_code
  • Performance metrics are self-reported by the official team; the model card contains only charts without itemized numerical data; this entry has not been independently verified

Open

Hugging Face Trending

Explore the latest AI Radar updates