Using a single video model for three types of conditional image generation: treating the input image (reference subject, Canny edge map, or original image to be edited) as an earlier frame in a video, and the target image to be generated as a subsequent frame, allowing the "inter-frame change" priors of the pre-trained video model to complete the generation. The authors observed that the relationship between input and target varies significantly across different tasks, so they designed two components: Mixture of Temporal Experts (MoTE), which dynamically selects temporal positional encoding experts with different "virtual time intervals" based on the input to determine how far the target is from the input; and Chain-of-Thought Reasoning for Time (CoTR), which inserts a latent intermediate state between the two frames and uses bottleneck-style attention masks to route information through it.
Likes 112 · Code Stars 3
Reasons
- A single model covers three types of tasks: subject-driven generation, Canny-to-image generation, and instruction-based image editing; the authors claim it outperforms compared methods on DreamBench, COCO2017, and GEdit-Bench.
- Ready to run: The repository provides inference code (the final configuration with three experts + one inference latent variable), LoRA weights are available on Hugging Face at lyneeeeeeeee/FrameForge, and there is a complete command for each of the three tasks.
- Transferable approach: Instead of training another image editing model, it places conditions and targets at "adaptive positions" on the video timeline, directly reusing the priors of the video foundation model; it ranked first in daily likes on Hugging Face (indicating attention only).
Usage
- Environment: Linux, Python 3.10, CUDA GPU; first install PyTorch corresponding to your CUDA version, then run pip install -r requirements.txt.
- Base models and encoders: Download tencent/HunyuanVideo-1.5, Qwen/Qwen2.5-VL-7B-Instruct, google/byt5-small via hf download, Glyph-SDXL-v2 via modelscope, and black-forest-labs/FLUX.1-Redux-dev (a gated repository requiring application on Hugging Face) into the ckpts/ directory.
- LoRA: Run hf download lyneeeeeeeee/FrameForge pytorch_lora_weights.safetensors --local-dir ./checkpoints.
- Inference: MODEL_PATH=./ckpts bash scripts/infer.sh --lora_path … --input_image input_image --reference_size 512 --prompt "…" --num_inference_steps 50; for the Canny task, the paper uses --guidance_scale 2.0, while the default for editing is 6.0. Both input and output are 512×512.
Risks
- Heavy dependencies: Requires the full HunyuanVideo 1.5 plus multiple encoders such as Qwen2.5-VL-7B; FLUX.1-Redux-dev is a gated model; the README does not specify VRAM requirements.
- Only inference code is released; training code and data are not included; output is fixed at 512×512 with 50 steps, and speed is not specified.
- Code and weights follow the Tencent Hunyuan Community License; review its terms before distribution or commercial use; performance comparisons are reported by the authors, and since generation results have not been reviewed, image quality is not evaluated.
Editor's Note: Suitable for research or workflow experiments aiming to use a single model for both "preserving subject while changing scene" and "instruction-based image editing," but first confirm that your GPU can accommodate HunyuanVideo 1.5.