VoxLock: Target-Speaker Extraction for Voice Agents Without an Enrollment Step

Voice agent transcribers often capture speech from everyone near the microphone. VoxLock is a 27M-parameter causal target-speaker extraction model, fine-tuned from the public REAL-TSE baseline on a single GPU. It extracts a sample of the caller's voice from the call itself (by default, the first 15 seconds; in streaming mode, it begins after hearing 3 seconds), then retains only that person's voice, removing speech from others, before passing it to speech recognition.

Likes 3 · GitHub Stars 88

Reasons

• The authors report reducing the word error rate from 23.21% to 4.95% on Krisp's 265 real-world recordings, with no significant difference compared to Krisp's best paid filtering; on AMI meeting windows not used in training, it reduced the error rate from 42.95% to 10.63%.
• Lightweight and real-time capable: the 8-bit model occupies 77% on a single CPU thread to handle one call, with an algorithmic latency of 24 ms; ONNX supports multiple precisions including int8, fp16, and fp32.
• Easy integration: pip install voxlock includes a Pipecat audio filter and a LiveKit Agents plugin; built-in support starts with sherpa-onnx 2.0.1. The model, code, and per-file scores are open-source under Apache-2.0.

Usage

• Command line: pip install voxlock → voxlock call.wav -o caller.wav (the model is downloaded automatically on first run).
• Python streaming: vl = voxlock.VoxLock(threads=1); stream = vl.live(). Push each 16 kHz audio chunk via stream.push(chunk) and call stream.flush() when finished. If a caller sample is already available, use vl.stream(sample) to begin isolation from the first segment.
• Voice agents: After pip install "voxlock[pipecat]", set VoxLockFilter() as the audio_in_filter. For LiveKit, use "voxlock[livekit]" and pass it as noise_cancellation.
• Try it in the browser: Hugging Face Space arbazsiddiqui/voxlock.

Risks

• Training data is English (LibriSpeech, AMI), and the model card lists the language as English; performance for Chinese and other languages has not been reported.
• The first few seconds rely on automatically identifying the caller at the start of the call. If someone else is speaking at the beginning, it may lock onto the wrong person; this can be mitigated by manually passing a sample. Comparisons with commercial products were conducted by the authors themselves.
• This is a personal author project. Evaluation scripts and per-file results are available in the repository's bench/ directory for independent verification.

Open · Code · Hugging Face Paper Page