Open-H3-Regenerate: Regenerate the H3 draft video to 4K

Video regeneration tool released by Efficient-Large-Model (code in the NVlabs/Sana repository): Use the 768P draft generated by MiniMax-H3 as a reference video in the context of the Ref2VA transformer, then use a Context (IC) LoRA to generate the 1080P, 2K (2560×1440) or 4K (3840×2160) version of the same segment, preserving faces, lip-sync, text, and layout from the draft. By default, it first upscales the draft using a latent space amplifier, then runs the last 3 steps of the HyperFlow 8-step grid; you can also choose 4 steps, or run the full 8 steps from noise, all with CFG 1. The repository includes inference code, download scripts, and a script to merge LoRAs, facilitating environments that can only load a single LoRA.

Original title: Efficient-Large-Model/Open-H3-Regenerate-2K-IC

Video regeneration tool released by Efficient-Large-Model (code in the NVlabs/Sana repository): Use the 768P draft generated by MiniMax-H3 as a reference video in the context of the Ref2VA transformer, then use a Context (IC) LoRA to generate the 1080P, 2K (2560×1440) or 4K (3840×2160) version of the same segment, preserving faces, lip-sync, text, and layout from the draft. By default, it first upscales the draft using a latent space amplifier, then runs the last 3 steps of the HyperFlow 8-step grid; you can also choose 4 steps, or run the full 8 steps from noise, all with CFG 1. The repository includes inference code, download scripts, and a script to merge LoRAs, facilitating environments that can only load a single LoRA.

Why it matters You can quickly test shots at low resolution first, then produce the high-definition final version once satisfied, without changing the visual layout or lip-sync

Upvotes 60 · Likes 60 · Downloads 11312

Reasons

  • Suitable for the workflow of "trial and error at low resolution, then final output at high resolution"; the author states that you can simply reuse the original prompt used to generate the draft, or leave it empty
  • The repository packages the HyperFlow 8-step LoRA, LMS Sharp sharpening LoRA, latent space amplifier, and fast VAE decoder, all fetched at once via the download script

Usage

  • After running pip install -r infer/requirements.txt, run python infer/download.py --out weights, then use torchrun to run infer.py, specifying the draft video with --input and filling in the original prompt with --caption
  • Choose 2k (default), 1080p, or 4k for --resolution; choose up3 (default), up4, or hf8 for --sampler; pass a jsonl file to --jobs for batch processing

Risks

  • The example command uses 8 GPUs; the transformer can be sharded across 1, 2, 4, 7, or 8 cards, but the author has not specified the single-GPU VRAM requirement
  • The MiniMax H3 community license does not cover the EU, UK, South Korea, and the US; this is not an official MiniMax product, and third-party components like LMS Sharp have separate terms
  • This entry has not been run; refer to the project page for actual results

Open

Hugging Face Trending

Xem tin AI mới nhất