Given an audio input mixed with multiple speakers, noise, and reverberation, along with the target speaker's lip movements, the model outputs only that person's voice in a causal manner, allowing for real-time streaming. The separator performs multi-scale time-frequency separation in a single encode-decode pass, unlike previous lightweight solutions that repeatedly invoke small separators. During training, an "encoder-space speaker contrastive loss" was added to specifically suppress crosstalk from nearby voices, with zero additional overhead during inference. The authors also release the RealSSA benchmark: simulated scenarios include 1–5 near-field interferers and 10–15 far-field interferers, alongside a real-recording test set.
Likes 149 · Code Stars 20
Reasons
- Comprehensive availability: Training/evaluation code, weights for both Falcon and Falcon-Big (hosted on Hugging Face as JusperLee/Falcon and JusperLee/Falcon-Big), a Hugging Face ZeroGPU online demo, and a local real-time demo called Falcon Studio. Both code and weights are licensed under Apache-2.0.
- The authors claim that compared to the causal AV-TFGridNet, it has 91.9% fewer parameters, an 20.2x reduction in computational cost, and a 11.1x reduction in GPU latency; the paper reports 0.5M trainable parameters (totaling 11.72M including the frozen lip-reading encoder).
- The authors state it achieves the best performance among causal methods on RealSSA-Sim, LRS2-2Mix, LRS3-2Mix, and VoxCeleb2-2Mix, with the README providing a full comparison table against nine baselines.
- Ready for deployment: Falcon Studio uses MediaPipe in the browser to track lips, sending the separated audio to headphones or via a virtual sound card to Zoom/Tencent Meeting; the README states that with a 2-second window and new audio output every 0.5 seconds, processing one window takes approximately 0.12 seconds on an Apple M5 Pro using Metal.
Usage
- Quickest experience: Open the Hugging Face Space linked in the README (JusperLee/Falcon-Studio), record a clip up to 60 seconds long in the browser, and separate it.
- Local real-time: git clone github.com/JusperLee/Falcon, pip install -r studio/requirements.txt (Python 3.10–3.12, supporting Apple Metal/NVIDIA CUDA/CPU). On macOS, double-click studio/Start Falcon.command or run studio/scripts/launch.sh; on Linux + NVIDIA, add --device cuda. Weights are automatically downloaded from Hugging Face on the first run.
- Offline single segment: Convert to 16 kHz mono mix.wav using ffmpeg, then run python infer_file.py --audio mix.wav --mouth mouth.npy --out target.wav --model JusperLee/Falcon; mouth.npy contains grayscale lip crops of the target person at 25fps and 96×96 resolution. The Python interface FalconRuntime.from_pretrained can also be used.
- Training: Organize data according to LRS2/LRS3 directory structures, build indices using scripts under DataPreProcess, and run python audio_train.py --conf_dir configs/Falcon.yml.
Risks
- Requires a frontal view/lip movements of the target speaker; it cannot be used in audio-only scenarios. Offline inference requires manually preparing the lip crop sequence.
- The virtual sound card solution for meeting integration (BlackHole) is for macOS; other systems require finding their own routing methods. The README does not provide complete instructions for real-time CPU execution.
- Both RealSSA datasets require manual approval; RealSSA-Sim is licensed under CC BY-NC-ND 4.0. The released multi-channel shards must be modified to work with the repository's mono training loader, and the README does not yet provide a link to the lip-reading visual backbone weights required for training.
- All quality and efficiency figures are reported by the authors; without hearing the demo audio, no assessment of actual listening quality is made.
Editor's note: Unlike standard noise reduction, this extracts the voice of "the person in the frame," removing other nearby speakers as well. It is suitable for meetings, interviews, and hearing aid applications.