AURA is an audio-driven streaming avatar model from Kyutai and Sorbonne University (paper arXiv 2611.26540): it distills a 50-step bidirectional teacher model into a causal student model that generates in segments of 4 steps, using a rolling KV cache to generate segment by segment, requiring only the audio for the current segment. The 1.3B version is based on Wan2.1-T2V-1.3B, uses Mimi audio encoding, has 576p, and is the only one in the family capable of real-time performance; there is also AURA-5B based on Snap TalkVerse (720p, the paper claims 9.7 frames per second on a single H100). Both provide merged standalone weights and training checkpoints containing only LoRA, and the training trajectory dataset is also public.
Why it matters It outputs video while receiving audio in small segments, without waiting for the entire speech; the paper claims end-to-end 20.7 frames per second on a single H100, which is 23 times faster than the teacher model.
Hugging Face likes 7 · Hugging Face trending score 7 · Hugging Face downloads 0
License: Non-commercial · ComfyUI: None · VRAM: Not specified
Reasons
- Includes paper, code repository, training configuration, and training metrics logs, making it reproducible
- The model card includes samples of the same speech paired with three different portraits
Usage
- Clone kyutai-labs/aura and run uv sync, download stage2_dmd/step-6500_ema_merged.safetensors (approx. 6.3 GB), then obtain the VAE and T5 text encoder from Wan2.1-T2V-1.3B, and follow the code repository instructions to input a portrait and a speech segment
Risks
- 1.3B is under CC BY-NC 4.0 and 5B is under Snap's non-commercial license, neither of which allows commercial use
- It can turn any portrait into a talking video; use only your own or authorized portraits, and label as AI-generated when publishing
- The real-time figures are measured by the paper on H100; the speed and VRAM requirements on consumer-grade GPUs are not specified in the model card; ComfyUI support is currently unavailable