PAN 2 is a general-purpose world model open-sourced by the Institute of Foundation Models (IFM) at MBZUAI: it takes the current frame (optionally an image) and a text-described action as input, and outputs a resulting video up to 20 seconds long (704×1248, 24 fps). A language model first converts the observations and actions into a structured description with a timeline, which is then used to generate the video via an MMDiT trained from scratch with 290 billion parameters. The text encoder uses Qwen3.5-9B, and the VAE uses Wan 2.2. The Comfy-Org repository contains files repackaged for ComfyUI, with the diffusion model available in BF16 and INT8 precisions.
Why it matters By having a large model first convert actions into a structured world state, and then having a diffusion model render it, agents can simulate before acting; ComfyUI packaged files are already available.
Hugging Face Likes 14 · Hugging Face Trending Score 14 · Hugging Face Downloads 0
License: Commercial use allowed · ComfyUI: Official packaging · VRAM: ≈106 GB estimated
Reasons
- Official Apache-2.0 license; the model card clearly documents the architecture, default sampling parameters, known limitations, and safety guardrails
- Official support for both Diffusers and vLLM; Comfy-Org provides the directory placement for each ComfyUI file
Usage
- ComfyUI: Download pan2_int8_convrot (or pan2_bf16), qwen3.5_9b_int8_convrot, and wan2.2_vae from Comfy-Org/PAN2, and place them in the diffusion_models, text_encoders, and vae directories under models, respectively
- Official script: After running hf download IFM/PAN2, convert the weights according to the model card and generate using inference/inference.sh; it is best to first use the official prompt rewriting skill to organize prompts into structured descriptions
Risks
- The official script loads the full 290 billion parameter diffusion model for each process; the 44B / approx. 106 GB estimate based on repository files differs from the official figures, so defer to the official numbers. For consumer-grade GPUs, the INT8 version is recommended
- The official documentation states that at least one step in about two-thirds of multi-step plans is not executed, and the model is sensitive to prompt wording; it is recommended to use the official pan2-guardrail content filter in conjunction