PAN 2: A world model that simulates video results based on text-described actions

A general-purpose world model released by the Institute for Foundation Models (IFM) at Mohamed bin Zayed University of Artificial Intelligence (MBZUAI): Given a current observation (optionally an image plus text) and actions described in text, it simulates the outcome of those actions, outputting videos up to 20 seconds long. It consists of two layers: a large language model performs "conceptual reasoning," writing scenes and actions into timestamped structured descriptions (events, actors, camera movements); an MMDiT with 290 billion parameters, trained from scratch, performs "perceptual reasoning," generating video latents based on these descriptions, which are then decoded by the Wan 2.2 VAE. A single checkpoint supports both text-to-video and image-to-video generation, outputting 704×1248 resolution at 24 fps, with default settings of 50 steps, CFG 2.5, and approximately 10 seconds.

Original Title: IFM/PAN2

A general-purpose world model released by the Institute for Foundation Models (IFM) at Mohamed bin Zayed University of Artificial Intelligence (MBZUAI): Given a current observation (optionally an image plus text) and actions described in text, it simulates the outcome of those actions, outputting videos up to 20 seconds long. It consists of two layers: a large language model performs "conceptual reasoning," writing scenes and actions into timestamped structured descriptions (events, actors, camera movements); an MMDiT with 290 billion parameters, trained from scratch, performs "perceptual reasoning," generating video latents based on these descriptions, which are then decoded by the Wan 2.2 VAE. A single checkpoint supports both text-to-video and image-to-video generation, outputting 704×1248 resolution at 24 fps, with default settings of 50 steps, CFG 2.5, and approximately 10 seconds.

Why It Matters Enables agents to simulate the results of each candidate plan before selecting the best one. Weights are Apache licensed, and support is available in both diffusers and vLLM.

Upvotes 70 · Likes 70 · Downloads 10

Reasons

  • The purpose is "simulation-based reasoning": agents or robots can use it to simulate each candidate plan first, then select the plan based on the simulation results; the model card also lists video generation uses for education, creation, etc.
  • Prompt rewriting tools are available: Officially provided "conceptual reasoning" skills for Codex / Claude Code and local rewriting scripts convert free text into the structured descriptions required by the model.
  • Deployment paths: The repository includes generate.py and multi-GPU scripts, and supports diffusers and vLLM; officially, pan2-guardrail safety filtering is also provided, reviewing inputs before and after generation.

Usage

  • Run hf download IFM/PAN2 --include "transformer/\*", then convert once using scripts/convert\_checkpoint.py to-native; additionally download Qwen/Qwen3.5-9B (text encoder) and Wan-AI/Wan2.2-TI2V-5B-Diffusers (VAE).
  • Run bash inference/inference.sh <checkpoint_path> <output_dir> [prompt JSON]; entries with image\_path are processed as image-to-video. Duration, steps, and CFG can be modified via environment variables such as VIDEO_LENGTH, NUM_STEPS, and GUIDANCE_SCALE.
  • For free-text prompts, first rewrite them into structured descriptions using the official pan-prompt-rewriter skill or inference/prompts\_rewritter.py before generating.

Risks

  • Each GPU must accommodate the full 290 billion parameter diffusion model; the official environment uses CUDA 13 and a preview version of flash-attn 4, resulting in high hardware requirements.
  • The official documentation notes that long plans often fail: approximately two-thirds of simulated plans miss at least one step; the model is sensitive to prompt wording, appearance drifts across segments, and physical constraints are sometimes ignored.
  • It can generate realistic videos; the official guidelines require labeling as AI-generated and using it in conjunction with guardrails; guardrails may also falsely block some normal requests.
  • Evaluation scores are from the official technical report; this entry has not been run, so actual performance should be verified against the model card.

Open

Hugging Face Trending

Explore the latest AI Radar updates