# VAD bake-off: FireRedVAD vs Silero vs pyannote (monsum-speech)

Workspace: /mnt/data01/Library/bakeoff-vad/ (scripts/, out/, data/hard/, venv/, src/ = FireRedVAD clone). Measure only; no monsum-speech source edited, nothing committed.

## Verdict
- FireRedVAD (default, thr 0.4) beats Silero (service defaults) by 0.36 WER points abs (20.98 vs 21.34 clean; 20.96 vs 21.30 hard), i.e. ~260 fewer errors of 71,349 ref words, mostly fewer deletions (3690 vs 4019). Single run, 8 files, one model, no CI: small, probably real direction, not proof.
- Vendor claim (FA 2.7% vs 9.4%) NOT reproduced as a FA gap: frame FA is higher for FireRed (5.1%) than Silero (4.0%), and miss is higher (7.4% vs 4.7%). Its ASR win comes from different span shape (longer spans, 8.2k vs 10.1k) not better frame accuracy against our coarse reference.
- The "hard" variant did not separate anything: no VAD was fooled by the FSD50K fill, ASR hallucination insertions did not rise. Real long-form recordings with untouched gaps were not found, so hallucination robustness is still untested. Integration is cheap (2 MB model, CPU) but it is a +0.36 pt win with a 3.3 GB CPU RAM peak on 94 min unless chunked; decide whether that is worth a new backend.

## Setup
- Eval: /mnt/data01/Library/datasets/speech/eval-diar, 8 files, 2785-5631 s, 21.6k s total, 71,349 ref words.
- Real long-form with speech timestamps: none found in /mnt/data01/Library/datasets/speech (only segment-cut corpora; the original recordings of monsum20261029 are not on disk).
- clean variant = eval-diar as is (gaps are digital silence).
- hard variant (data/hard/*.wav, scripts/build_hard.py, seed 0): every gap between union of reference spans (incl. leading/trailing) replaced by randomly concatenated non-speech clips from FSD50K eval split (/mnt/data01/Library/datasets/sound-event/FSD50K), each clip RMS-normalised then scaled to a per-gap level uniform in -35..-20 dB re the file's speech-active RMS, 10 ms fades. Excluded labels: any voice/speech/singing/chatter/crowd/laughter/cough/breathing/animal/alarm/explosion etc. Used: music, instruments, doors, footsteps, typing-like, engines, water, wind, hiss, etc (build_log.json lists labels). 6113 usable clips. Speech regions untouched (noise only in gaps). Caveat: gaps are short (median file has 100-380 gaps, only 20-30% over 2 s) and noise sits at modest level, so this is a mild stress; no noise under speech.
- Frame metrics: 10 ms frames vs union of reference segment spans. Miss % = ref-speech frames not detected / ref-speech frames. FA % = detected frames outside ref / non-ref frames. CAVEAT: reference spans are human segment boundaries including internal pauses, so "miss" is inflated for any VAD that cuts pauses (pyannote especially), and "FA" includes VAD padding at span edges.
- ASR: monsum-speech `asr.transcribe` in-process (cwd monsum-speech, .env: ct2-whisper-large-v3-turbo-mn-1, language=mn, batch 4, GPU shared). Each VAD's spans are injected via a stub of faster_whisper `get_speech_timestamps` with `vad_filter=True`, so every VAD gets the same packing into <=15 s chunks as production Silero. Spans >15 s split evenly (only matters for pyannote; Silero capped at 15 s as production does, FireRed max_speech_frame=1500 instead of documented 2000 to fit the 15 s chunk). I first tried `vad_segments`/clip_timestamps (one chunk per span, ~3x slower, no packing) and aborted it. No diarization. Scoring: mn_text.normalize_text + jiwer, WER over time-ordered concatenated text (plain WER of eval_speech.py), micro-averaged. Sanity: silero_default clean 21.34 vs cached baseline hyp/lidfix 21.37 (WER 0.2137).
- VAD configs: silero_default = monsum service defaults (thr 0.5, min_silence 0.07, pad 0.2, max 15 s); silero_thr0.7; firered_default = README defaults (thr 0.4, smooth 5, min_speech 20 fr, min_silence 20 fr; use_gpu False); firered_thr0.6; pyannote_default = monsum `pyannote` backend defaults, CPU. Thresholds chosen a priori (stricter variant), nothing tuned on this data.

## Results: clean variant (digital-silence gaps)
| VAD | miss % | FA % | spans | median span s | WER % | ins | del | sub |
|---|---|---|---|---|---|---|---|---|
| silero default (0.5) | 4.73 | 4.01 | 10081 | 1.82 | 21.34 | 2264 | 4019 | 8946 |
| silero thr 0.7 | 7.20 | 3.25 | 12133 | 1.55 | 22.16 | 2214 | 4537 | 9058 |
| FireRed default (0.4) | 7.37 | 5.06 | 8183 | 2.22 | **20.98** | 2324 | 3690 | 8957 |
| FireRed thr 0.6 | 9.17 | 4.24 | 9090 | 1.99 | 21.41 | 2232 | 3924 | 9119 |
| pyannote default | 14.36 | 0.92 | 8371 | 1.79 | 22.23 | 2194 | 4328 | 9337 |

## Results: hard variant (FSD50K non-speech in gaps)
| VAD | miss % | FA % | spans | median span s | WER % | ins | del | sub |
|---|---|---|---|---|---|---|---|---|
| silero default (0.5) | 4.61 | 4.44 | 10154 | 1.80 | 21.30 | 2277 | 3972 | 8949 |
| silero thr 0.7 | 7.07 | 3.46 | 12247 | 1.54 | 22.06 | 2213 | 4456 | 9071 |
| FireRed default (0.4) | 7.36 | 5.71 | 8188 | 2.22 | **20.96** | 2314 | 3634 | 9008 |
| FireRed thr 0.6 | 9.15 | 4.61 | 9093 | 1.99 | 21.51 | 2243 | 3933 | 9172 |
| pyannote default | 14.35 | 2.30 | 8383 | 1.79 | 22.24 | 2192 | 4335 | 9344 |

Clean vs hard barely moves: FA rises 0.4-1.4 pts (pyannote 0.9 -> 2.3, FireRed 5.1 -> 5.7, Silero 4.0 -> 4.4) but insertions are flat (+-15), so ASR is not hallucinating on this noise. Insertions (~2.2-2.3k) are ordinary ASR errors, not VAD-driven. Silero default vs FireRed default per-file WER differs mostly on 01a0* files (e.g. 33.1/33.9 vs 32.5/32.1 on the two worst); others tie.

## Cost (CPU only, torch threads 4; GPU: none used by any VAD)
Longest file 01a151d2 = 5631 s (94 min), clean, quiet machine (load ~1), 2 repeats each:
| VAD | wall s (94 min) | RTF | process peak RSS | RSS after load |
|---|---|---|---|---|
| Silero (faster-whisper ONNX, 1 thread) | 6.4 / 6.4 | 0.0011 | 1987 MB | 964 MB (includes monsum_speech+torch imports) |
| FireRedVAD (torch CPU, 4 thr) | 10.9 / 10.9 | 0.0019 | 3751 MB | 453 MB |
| pyannote seg (CPU, 4 thr) | 56.2 / 56.1 | 0.010 | 1917 MB | 1070 MB |
Peak RSS is process-wide ru_maxrss, so Silero/pyannote figures include the monsum import baseline; FireRed's delta is ~+3.3 GB on this file (4.8 GB cumulative over all 8 files) because the whole file's fbank/activations are held; model itself is 2.3 MB. Walls in out/frame_metrics.json from the sweep were taken under heavy CPU load from the other bake-off and are not comparable; use the table above.

## License / provenance
FireRedVAD: Apache-2.0 (GitHub LICENSE, pyproject, HF model card tag license:apache-2.0), i.e. commercial use allowed, not non-commercial. Repo FireRedTeam/FireRedVAD commit c30ec49e, package fireredvad 0.0.2; weights HF FireRedTeam/FireRedVAD (VAD/, Stream-VAD/, AED/; ONNX copies in repo pretrained_models/onnx_models). Paper arXiv 2603.10420. Silero via faster-whisper (MIT). pyannote segmentation as shipped in monsum models/.

## Install notes
`uv venv --python 3.12 venv; uv pip install torch --index-url https://download.pytorch.org/whl/cpu; uv pip install -e src numpy soundfile huggingface_hub` then `snapshot_download('FireRedTeam/FireRedVAD', local_dir='models/FireRedVAD')`. Works with torch 2.15 CPU / numpy 2.5 despite repo requirements.txt pinning torch 2.1.0+cu118 / numpy 1.26.1. Deps: numpy, kaldiio, kaldi-native-fbank, soundfile, textgrid, torch. Emits "Too long input, split every 30000 frames" and "Unexpected short speech segment" messages (benign).
API: `FireRedVad.from_pretrained(dir, FireRedVadConfig(...)).detect(wav_path)` -> `{'timestamps': [(s,e),...]}`; takes a file path, not an array.

## What integrating FireRed would take
- `VadBackend.FIRERED = "firered"` in schemas.py; `_detect_firered` in adapters/vad.py returning VadSegment list; in pipeline.run_transcription treat it like PYANNOTE (own pass, spans handed to ASR; the `use_silero` condition already excludes it). Lazy import like pyannote.
- `resolve()` mapping: vad_threshold -> speech_threshold, min_speech_duration -> min_speech_frame (10 ms frames), min_silence_duration -> min_silence_frame, cap max_speech at CHUNK_LENGTH_S (15 s -> max_speech_frame 1500). speech_pad has no analogue (extend_speech_frame is the closest). New per-backend defaults (thr 0.4, 0.2 s / 0.2 s min speech/silence).
- Config: `VAD_FIREREDVAD_PATH` for the 2.3 MB model dir (bundle in models/, HF_HUB_OFFLINE-safe).
- Deps: fireredvad (pip from git/clone, no PyPI release verified), kaldiio, kaldi-native-fbank, textgrid; torch already present. CPU is fine (10.9 s per 94 min); needs `detect` to accept a waveform or the adapter to write a temp wav; chunk long audio (e.g. 10-15 min windows) to avoid the ~3.3 GB RSS spike. ONNX variant exists (onnxruntime already a faster-whisper dependency) and would likely avoid torch memory; not tested.
- Also noted: caller-supplied `vad_segments` through clip_timestamps are not packed and truncate at 30 s per span; spans from any non-Silero backend should be split to <=15 s (pyannote already hits this).

## Not done / limits
- No real unstitched recordings; hard variant is mild; no noise under speech.
- One model, one run per config, no confidence intervals; 0.36 pt gap not significance-tested.
- Silero production path with its own span packing (`vad_filter=True`, no external spans) not run (dropped for time); cached lidfix baseline 21.37 matches the injected-spans silero_default, supporting equivalence.
- GPU was shared and CPU heavily loaded during the ASR sweep (several OOM-retries; resumed from per-file cache, no result mixing); ASR wall times in out/asr_scores.json are not meaningful.