# LID bake-off: AmberNet vs SpeechBrain ECAPA vs MMS-LID vs Whisper Date 2026-11-02. Offline, CPU unless noted. No monsum-speech source edited, nothing committed. Workspace `/mnt/data01/Library/bakeoff-lid/` (scripts `run.py`, `analyze.py`, `summary.py`, `bench.py`; raw per-clip log-probs in `out/*.pkl`; `env.sh`). ## Verdict 1. Keep AmberNet. Change config only: `LID_LANGUAGES=mn,en,ru,zh,kk`, `LID_MIN_PROB=0.9`, `LID_MIN_SECONDS=3` (the code default; `.env` currently has 1 and 0.7). No other model beats it on both axes, and all are 5x to 40x slower on CPU. 2. Restricting labels is the big win: foreign recall at 3 s jumps (AmberNet en 0.62 -> 0.98, ru 0.57 -> 0.97, kk 0.57 -> 0.75) with equal or lower Mongolian false-foreign. Cost: below 2 s the 5-way softmax collapses Mongolian to `en` (62/100 FLEURS mn 1 s crops), so `LID_MIN_SECONDS` must stay >= 3 (2 s gives 5% to 9% mn false-foreign even at 0.9/0.7). 3. All models give 0/108 false-foreign on 30 s request clips at every threshold, so `_resolve_model` routing is safe with any of them. Foreign recall at 30 s was NOT measured (no foreign 30 s data); 5 s recall of 0.9 to 1.0 suggests it is fine. ## Data and method - FLEURS crops: `/mnt/data01/Library/datasets/speech/eval-lid`, 1993 clips (100 per language per crop length 1/2/3/5 s; en 94 and zh 99 at 5 s). Crops start at clip start (leading silence lowers 1 s numbers). I did not use `score-lid` (it calls the HTTP service); I scored raw model outputs and applied `language_rule._judge` semantics myself: leave `mn` only if top != mn, top is Whisper-decodable, and p(top) >= LID_MIN_PROB. Single clip, no speaker pooling, `LID_MIN_SECONDS` not applied (tables are per crop length so you can pick it). - In-domain Mongolian: 1000 segments from `monsum20261029`, 200 per duration bucket (1-2, 2-3, 3-5, 5-10, 10+ s, 10+ cut to 30 s), round-robin over 25 sources and many speakers (117 to 145 speakers per bucket), 24 kHz -> 16 kHz. Code-switch filter (>50% of letters Latin/Han) skipped 0 of 8422 segments: transcripts are all Cyrillic. But English speech transliterated into Cyrillic exists in the data (see caveat), so in-domain false-foreign is an upper bound. - 30 s request clips: 108 (25 sources first 30 s, 75 random 30 s windows from the same sources, 8 eval-diar files first 30 s of speech from the reference spans; segments > 15 s dropped to avoid super-segments). Eval-diar files come from the same sources, so these are not independent of the 25. - AmberNet run through the repo adapter (`adapters/lid.py::probabilities`, py3.12 venv, `PYTHONPATH=monsum-speech/src`) twice: `LID_LANGUAGES` empty and `mn,en,ru,zh,kk`. Offline renormalization of the full run matches the adapter's restricted run to 2.4e-7 (max abs diff), so "r5" for the other models is renormalization over their 5 labels (MMS mn = `mon`+`khk` summed, zh = `cmn`, kk = `kaz`; SpeechBrain/Whisper use ISO-2 codes). - "full" = argmax over all labels; a non-mn top counts as a switch only if Whisper can decode it (list in `analyze.py`; not applied to MMS 3-letter codes, which slightly overstates MMS full false-foreign). "r5" = restricted to the 5. - MMS-LID ran on GPU fp16 batch 1 (CPU fp32 was ~3 s per 3 s clip under host load 30+, which would have taken hours). fp16 vs fp32 agreement not checked. Whisper = local `ct2-whisper-large-v3-turbo` (base, not the Mongolian fine-tune), faster-whisper `detect_language`, CPU int8, reference only, on a small subset (15 clips per language per length = 298 FLEURS, 50 in-domain, no 30 s) so its numbers are coarse (n=15 per cell). - Statistical resolution: FLEURS cells n=100 (95% CI about +-3 to 10 points), in-domain buckets n=200 (+-1 to 4). Differences under about 3 points between models are noise. Nothing was tuned on test except the threshold sweep. ## Results #### A. FLEURS top-1 accuracy (raw argmax) | config | n | 1s mn | 1s others(en/ru/zh/kk) | 2s mn | 2s others(en/ru/zh/kk) | 3s mn | 3s others(en/ru/zh/kk) | 5s mn | 5s others(en/ru/zh/kk) | |:------------------|-----:|--------:|:-------------------------|--------:|:-------------------------|--------:|:-------------------------|--------:|:-------------------------| | ambernet full | 1993 | 0.05 | 0.05/0.15/0.02/0.05 | 0.6 | 0.65/0.46/0.48/0.28 | 0.9 | 0.84/0.69/0.75/0.64 | 0.99 | 0.97/0.91/0.94/0.93 | | ambernet r5 | 1993 | 0.18 | 0.61/0.40/0.15/0.12 | 0.8 | 0.93/0.80/0.63/0.44 | 0.96 | 0.99/0.97/0.76/0.80 | 1 | 1.00/1.00/0.97/0.98 | | speechbrain full | 1993 | 0.08 | 0.11/0.08/0.04/0.04 | 0.66 | 0.73/0.34/0.40/0.32 | 0.97 | 0.95/0.52/0.72/0.65 | 0.99 | 0.99/0.60/0.93/0.90 | | speechbrain r5 | 1993 | 0.28 | 0.66/0.40/0.19/0.15 | 0.87 | 0.96/0.78/0.64/0.51 | 0.99 | 1.00/0.97/0.79/0.79 | 0.99 | 1.00/1.00/0.96/0.94 | | mms-lid-126 full | 1993 | 0.05 | 0.22/0.05/0.12/0.04 | 0.66 | 0.90/0.54/0.55/0.33 | 0.97 | 0.98/0.78/0.77/0.65 | 0.99 | 1.00/0.96/0.97/0.93 | | mms-lid-126 r5 | 1993 | 0.08 | 0.84/0.29/0.19/0.13 | 0.76 | 0.97/0.75/0.61/0.49 | 0.98 | 1.00/0.95/0.82/0.79 | 0.99 | 1.00/1.00/0.97/0.94 | | mms-lid-256 full | 1993 | 0.05 | 0.21/0.05/0.10/0.05 | 0.61 | 0.89/0.45/0.52/0.31 | 0.94 | 0.96/0.77/0.80/0.66 | 1 | 1.00/0.96/0.97/0.92 | | mms-lid-256 r5 | 1993 | 0.07 | 0.80/0.24/0.20/0.12 | 0.69 | 0.98/0.72/0.65/0.48 | 0.97 | 1.00/0.96/0.84/0.84 | 1 | 1.00/1.00/0.97/0.97 | | mms-lid-4017 full | 1993 | 0.05 | 0.24/0.09/0.12/0.01 | 0.68 | 0.89/0.46/0.58/0.26 | 0.95 | 0.97/0.75/0.78/0.63 | 1 | 1.00/0.94/0.96/0.86 | | mms-lid-4017 r5 | 1993 | 0.08 | 0.81/0.23/0.18/0.04 | 0.85 | 0.99/0.76/0.64/0.37 | 0.99 | 1.00/0.95/0.83/0.80 | 1 | 1.00/1.00/0.97/0.94 | | whisper full | 298 | 0 | 1.00/0.33/0.27/0.00 | 0 | 1.00/0.53/0.60/0.20 | 0 | 1.00/0.87/0.73/0.60 | 0 | 1.00/1.00/1.00/0.93 | | whisper r5 | 298 | 0 | 1.00/0.33/0.27/0.00 | 0 | 1.00/0.60/0.60/0.27 | 0 | 1.00/0.93/0.87/0.60 | 0 | 1.00/1.00/1.00/0.93 | #### B. Mongolian false-foreign rate, % (F-mn = FLEURS mn crops, n=100/length; ID = in-domain monsum20261029, n=200/bucket, 30 s req = 108 clips) | config | thr | F-mn 2s | F-mn 3s | F-mn 5s | ID 1-2s | ID 2-3s | ID 3-5s | ID 5-10s | ID 10-infs | 30s req | |:------------------|------:|----------:|----------:|----------:|----------:|----------:|----------:|-----------:|-------------:|:----------| | ambernet full | 0.5 | 25 | 6 | 1 | 14 | 4.5 | 5.5 | 0.5 | 0.5 | 0/108 | | ambernet full | 0.7 | 13 | 3 | 0 | 7 | 2 | 3 | 0 | 0.5 | 0/108 | | ambernet full | 0.8 | 10 | 2 | 0 | 4.5 | 2 | 2 | 0 | 0.5 | 0/108 | | ambernet full | 0.9 | 8 | 1 | 0 | 2.5 | 1.5 | 0.5 | 0 | 0.5 | 0/108 | | ambernet r5 | 0.5 | 13 | 3 | 0 | 8 | 2.5 | 3.5 | 1.5 | 1 | 0/108 | | ambernet r5 | 0.7 | 9 | 2 | 0 | 5.5 | 2 | 2.5 | 1.5 | 1 | 0/108 | | ambernet r5 | 0.8 | 7 | 1 | 0 | 4 | 2 | 2.5 | 1 | 1 | 0/108 | | ambernet r5 | 0.9 | 5 | 1 | 0 | 3 | 1.5 | 2 | 1 | 0.5 | 0/108 | | speechbrain full | 0.5 | 7 | 3 | 1 | 15 | 6 | 4.5 | 1.5 | 0.5 | 0/108 | | speechbrain full | 0.7 | 3 | 2 | 0 | 6 | 3 | 2.5 | 0.5 | 0.5 | 0/108 | | speechbrain full | 0.8 | 2 | 1 | 0 | 4.5 | 0.5 | 2 | 0.5 | 0.5 | 0/108 | | speechbrain full | 0.9 | 1 | 1 | 0 | 1.5 | 0 | 1 | 0.5 | 0.5 | 0/108 | | speechbrain r5 | 0.5 | 10 | 1 | 1 | 7.5 | 4 | 4.5 | 1.5 | 1 | 0/108 | | speechbrain r5 | 0.7 | 7 | 1 | 1 | 4.5 | 2.5 | 3 | 1.5 | 1 | 0/108 | | speechbrain r5 | 0.8 | 4 | 1 | 0 | 4 | 2.5 | 2 | 1 | 1 | 0/108 | | speechbrain r5 | 0.9 | 2 | 0 | 0 | 1.5 | 2 | 2 | 1 | 1 | 0/108 | | mms-lid-126 full | 0.5 | 11 | 2 | 0 | 10 | 5 | 4 | 3 | 0.5 | 0/108 | | mms-lid-126 full | 0.7 | 4 | 2 | 0 | 5.5 | 4 | 2 | 2 | 0 | 0/108 | | mms-lid-126 full | 0.8 | 3 | 1 | 0 | 4 | 3.5 | 2 | 2 | 0 | 0/108 | | mms-lid-126 full | 0.9 | 2 | 1 | 0 | 2.5 | 2.5 | 1.5 | 1 | 0 | 0/108 | | mms-lid-126 r5 | 0.5 | 20 | 2 | 1 | 12 | 6 | 4 | 2 | 0.5 | 0/108 | | mms-lid-126 r5 | 0.7 | 10 | 2 | 0 | 4.5 | 3 | 4 | 1.5 | 0.5 | 0/108 | | mms-lid-126 r5 | 0.8 | 7 | 2 | 0 | 2 | 2 | 3.5 | 1.5 | 0.5 | 0/108 | | mms-lid-126 r5 | 0.9 | 3 | 1 | 0 | 1 | 2 | 2.5 | 1 | 0 | 0/108 | | mms-lid-256 full | 0.5 | 8 | 4 | 0 | 14 | 4 | 2.5 | 1 | 0 | 0/108 | | mms-lid-256 full | 0.7 | 3 | 4 | 0 | 8 | 3 | 2 | 0.5 | 0 | 0/108 | | mms-lid-256 full | 0.8 | 2 | 4 | 0 | 4.5 | 2.5 | 2 | 0.5 | 0 | 0/108 | | mms-lid-256 full | 0.9 | 2 | 3 | 0 | 3 | 2 | 1.5 | 0.5 | 0 | 0/108 | | mms-lid-256 r5 | 0.5 | 23 | 2 | 0 | 12 | 4.5 | 4.5 | 2 | 0.5 | 0/108 | | mms-lid-256 r5 | 0.7 | 15 | 2 | 0 | 6.5 | 3.5 | 3.5 | 1 | 0.5 | 0/108 | | mms-lid-256 r5 | 0.8 | 8 | 2 | 0 | 5 | 3 | 3.5 | 1 | 0.5 | 0/108 | | mms-lid-256 r5 | 0.9 | 5 | 1 | 0 | 3 | 2 | 2.5 | 1 | 0.5 | 0/108 | | mms-lid-4017 full | 0.5 | 3 | 3 | 0 | 13.5 | 6.5 | 4 | 1.5 | 0 | 0/108 | | mms-lid-4017 full | 0.7 | 2 | 3 | 0 | 6.5 | 4.5 | 3 | 1.5 | 0 | 0/108 | | mms-lid-4017 full | 0.8 | 2 | 2 | 0 | 5 | 4 | 2.5 | 1 | 0 | 0/108 | | mms-lid-4017 full | 0.9 | 2 | 1 | 0 | 3 | 3.5 | 2 | 0.5 | 0 | 0/108 | | mms-lid-4017 r5 | 0.5 | 11 | 1 | 0 | 11.5 | 4 | 5 | 1.5 | 0.5 | 0/108 | | mms-lid-4017 r5 | 0.7 | 4 | 1 | 0 | 7 | 3 | 3.5 | 1 | 0.5 | 0/108 | | mms-lid-4017 r5 | 0.8 | 2 | 0 | 0 | 3.5 | 2.5 | 3 | 1 | 0.5 | 0/108 | | mms-lid-4017 r5 | 0.9 | 2 | 0 | 0 | 2 | 2 | 3 | 1 | 0 | 0/108 | | whisper full | 0.5 | 33.3 | 60 | 46.7 | 30 | 10 | 30 | 0 | 0 | - | | whisper full | 0.7 | 13.3 | 26.7 | 6.7 | 0 | 0 | 10 | 0 | 0 | - | | whisper full | 0.8 | 6.7 | 20 | 6.7 | 0 | 0 | 10 | 0 | 0 | - | | whisper full | 0.9 | 6.7 | 0 | 0 | 0 | 0 | 10 | 0 | 0 | - | | whisper r5 | 0.5 | 73.3 | 73.3 | 93.3 | 60 | 70 | 90 | 90 | 50 | - | | whisper r5 | 0.7 | 46.7 | 53.3 | 86.7 | 30 | 20 | 60 | 40 | 20 | - | | whisper r5 | 0.8 | 26.7 | 40 | 60 | 20 | 20 | 60 | 10 | 10 | - | | whisper r5 | 0.9 | 13.3 | 33.3 | 33.3 | 10 | 0 | 50 | 10 | 10 | - | #### C. Foreign recall under the decision rule (FLEURS; switched to the correct language) | config | thr | 2s en/ru/zh/kk | 3s en/ru/zh/kk | 5s en/ru/zh/kk | |:------------------|------:|:--------------------|:--------------------|:--------------------| | ambernet full | 0.5 | 0.61/0.44/0.46/0.27 | 0.77/0.67/0.75/0.62 | 0.97/0.91/0.94/0.93 | | ambernet full | 0.7 | 0.47/0.38/0.39/0.22 | 0.66/0.60/0.75/0.59 | 0.88/0.82/0.92/0.89 | | ambernet full | 0.8 | 0.38/0.36/0.35/0.21 | 0.62/0.57/0.75/0.57 | 0.83/0.76/0.92/0.88 | | ambernet full | 0.9 | 0.29/0.31/0.35/0.20 | 0.50/0.49/0.75/0.52 | 0.70/0.64/0.92/0.85 | | ambernet r5 | 0.5 | 0.93/0.79/0.61/0.44 | 0.99/0.97/0.76/0.77 | 1.00/1.00/0.97/0.98 | | ambernet r5 | 0.7 | 0.90/0.77/0.58/0.41 | 0.98/0.97/0.76/0.75 | 1.00/1.00/0.97/0.97 | | ambernet r5 | 0.8 | 0.86/0.76/0.56/0.38 | 0.98/0.97/0.76/0.75 | 1.00/1.00/0.97/0.94 | | ambernet r5 | 0.9 | 0.85/0.73/0.56/0.35 | 0.96/0.96/0.76/0.73 | 1.00/1.00/0.97/0.90 | | speechbrain full | 0.5 | 0.62/0.33/0.39/0.29 | 0.89/0.49/0.72/0.57 | 0.99/0.58/0.93/0.89 | | speechbrain full | 0.7 | 0.50/0.27/0.37/0.26 | 0.84/0.41/0.72/0.50 | 0.97/0.44/0.90/0.82 | | speechbrain full | 0.8 | 0.43/0.20/0.37/0.20 | 0.79/0.32/0.70/0.47 | 0.96/0.35/0.89/0.78 | | speechbrain full | 0.9 | 0.36/0.10/0.32/0.18 | 0.64/0.23/0.67/0.35 | 0.90/0.23/0.89/0.72 | | speechbrain r5 | 0.5 | 0.96/0.78/0.64/0.48 | 1.00/0.97/0.78/0.76 | 1.00/1.00/0.96/0.94 | | speechbrain r5 | 0.7 | 0.95/0.76/0.63/0.46 | 1.00/0.96/0.77/0.74 | 1.00/1.00/0.95/0.91 | | speechbrain r5 | 0.8 | 0.94/0.74/0.62/0.42 | 1.00/0.93/0.76/0.68 | 1.00/1.00/0.95/0.90 | | speechbrain r5 | 0.9 | 0.92/0.70/0.58/0.40 | 1.00/0.93/0.74/0.64 | 1.00/1.00/0.93/0.88 | | mms-lid-126 full | 0.5 | 0.82/0.51/0.55/0.31 | 0.94/0.76/0.77/0.64 | 1.00/0.96/0.97/0.92 | | mms-lid-126 full | 0.7 | 0.78/0.46/0.49/0.26 | 0.93/0.74/0.76/0.63 | 0.99/0.95/0.96/0.91 | | mms-lid-126 full | 0.8 | 0.71/0.40/0.49/0.25 | 0.92/0.73/0.76/0.60 | 0.99/0.94/0.96/0.89 | | mms-lid-126 full | 0.9 | 0.61/0.33/0.41/0.20 | 0.88/0.67/0.76/0.51 | 0.96/0.92/0.96/0.87 | | mms-lid-126 r5 | 0.5 | 0.97/0.74/0.61/0.45 | 1.00/0.95/0.81/0.79 | 1.00/1.00/0.97/0.94 | | mms-lid-126 r5 | 0.7 | 0.95/0.72/0.61/0.43 | 0.99/0.94/0.80/0.74 | 1.00/1.00/0.97/0.93 | | mms-lid-126 r5 | 0.8 | 0.94/0.72/0.60/0.40 | 0.99/0.93/0.80/0.72 | 1.00/1.00/0.97/0.92 | | mms-lid-126 r5 | 0.9 | 0.92/0.69/0.60/0.37 | 0.99/0.92/0.80/0.71 | 1.00/1.00/0.97/0.89 | | mms-lid-256 full | 0.5 | 0.83/0.44/0.49/0.28 | 0.95/0.77/0.80/0.65 | 1.00/0.96/0.97/0.92 | | mms-lid-256 full | 0.7 | 0.77/0.39/0.46/0.25 | 0.92/0.68/0.78/0.63 | 1.00/0.88/0.97/0.92 | | mms-lid-256 full | 0.8 | 0.72/0.34/0.46/0.24 | 0.91/0.63/0.77/0.59 | 0.98/0.88/0.96/0.92 | | mms-lid-256 full | 0.9 | 0.66/0.29/0.42/0.20 | 0.90/0.54/0.71/0.56 | 0.98/0.76/0.96/0.88 | | mms-lid-256 r5 | 0.5 | 0.98/0.72/0.64/0.46 | 1.00/0.94/0.84/0.82 | 1.00/1.00/0.97/0.97 | | mms-lid-256 r5 | 0.7 | 0.95/0.68/0.61/0.44 | 0.98/0.94/0.84/0.80 | 1.00/1.00/0.97/0.96 | | mms-lid-256 r5 | 0.8 | 0.93/0.68/0.61/0.42 | 0.98/0.93/0.83/0.79 | 1.00/1.00/0.97/0.96 | | mms-lid-256 r5 | 0.9 | 0.92/0.65/0.58/0.41 | 0.98/0.93/0.82/0.75 | 1.00/1.00/0.97/0.96 | | mms-lid-4017 full | 0.5 | 0.79/0.43/0.54/0.24 | 0.97/0.75/0.77/0.57 | 0.99/0.94/0.95/0.84 | | mms-lid-4017 full | 0.7 | 0.74/0.34/0.51/0.21 | 0.95/0.68/0.76/0.53 | 0.99/0.90/0.95/0.77 | | mms-lid-4017 full | 0.8 | 0.68/0.34/0.51/0.20 | 0.93/0.65/0.76/0.51 | 0.98/0.88/0.95/0.76 | | mms-lid-4017 full | 0.9 | 0.66/0.27/0.50/0.19 | 0.89/0.53/0.75/0.47 | 0.98/0.70/0.95/0.75 | | mms-lid-4017 r5 | 0.5 | 0.96/0.74/0.64/0.35 | 1.00/0.95/0.81/0.77 | 1.00/1.00/0.97/0.93 | | mms-lid-4017 r5 | 0.7 | 0.96/0.73/0.63/0.35 | 1.00/0.94/0.81/0.73 | 1.00/1.00/0.97/0.93 | | mms-lid-4017 r5 | 0.8 | 0.96/0.73/0.61/0.34 | 1.00/0.93/0.81/0.72 | 1.00/1.00/0.97/0.91 | | mms-lid-4017 r5 | 0.9 | 0.95/0.68/0.60/0.33 | 1.00/0.93/0.80/0.70 | 1.00/1.00/0.97/0.91 | | whisper full | 0.5 | 1.00/0.47/0.60/0.20 | 1.00/0.87/0.73/0.53 | 1.00/1.00/1.00/0.93 | | whisper full | 0.7 | 1.00/0.47/0.60/0.20 | 1.00/0.87/0.73/0.47 | 1.00/1.00/1.00/0.93 | | whisper full | 0.8 | 1.00/0.47/0.60/0.07 | 1.00/0.87/0.73/0.47 | 1.00/1.00/1.00/0.93 | | whisper full | 0.9 | 1.00/0.47/0.53/0.07 | 1.00/0.87/0.73/0.47 | 1.00/1.00/1.00/0.93 | | whisper r5 | 0.5 | 1.00/0.60/0.60/0.27 | 1.00/0.93/0.80/0.60 | 1.00/1.00/1.00/0.93 | | whisper r5 | 0.7 | 1.00/0.60/0.60/0.20 | 1.00/0.93/0.73/0.53 | 1.00/1.00/1.00/0.93 | | whisper r5 | 0.8 | 1.00/0.60/0.60/0.13 | 1.00/0.93/0.73/0.47 | 1.00/1.00/1.00/0.93 | | whisper r5 | 0.9 | 1.00/0.53/0.53/0.07 | 1.00/0.93/0.73/0.47 | 1.00/1.00/1.00/0.93 | #### D. FLEURS Mongolian raw misses (counts out of 100) | config | 1s | 2s | 3s | 5s | |:------------------|:-------------------------|:-------------------------|:------------------------|:------------------------| | ambernet full | ru:0 kk:1 en:2 other:92 | ru:0 kk:1 en:2 other:37 | ru:0 kk:1 en:0 other:9 | ru:0 kk:0 en:0 other:1 | | ambernet r5 | ru:15 kk:5 en:62 other:0 | ru:6 kk:8 en:5 other:1 | ru:0 kk:3 en:1 other:0 | ru:0 kk:0 en:0 other:0 | | speechbrain full | ru:0 kk:1 en:1 other:90 | ru:0 kk:0 en:0 other:34 | ru:0 kk:1 en:0 other:2 | ru:0 kk:1 en:0 other:0 | | speechbrain r5 | ru:2 kk:6 en:63 other:1 | ru:0 kk:7 en:6 other:0 | ru:0 kk:1 en:0 other:0 | ru:0 kk:1 en:0 other:0 | | mms-lid-126 full | ru:1 kk:1 en:0 other:93 | ru:0 kk:4 en:0 other:30 | ru:0 kk:2 en:0 other:1 | ru:0 kk:0 en:0 other:1 | | mms-lid-126 r5 | ru:1 kk:2 en:88 other:1 | ru:0 kk:8 en:15 other:1 | ru:0 kk:2 en:0 other:0 | ru:0 kk:1 en:0 other:0 | | mms-lid-256 full | ru:0 kk:0 en:0 other:95 | ru:0 kk:5 en:0 other:34 | ru:0 kk:1 en:0 other:5 | ru:0 kk:0 en:0 other:0 | | mms-lid-256 r5 | ru:1 kk:3 en:87 other:2 | ru:1 kk:14 en:13 other:3 | ru:0 kk:3 en:0 other:0 | ru:0 kk:0 en:0 other:0 | | mms-lid-4017 full | ru:1 kk:0 en:0 other:94 | ru:0 kk:1 en:0 other:31 | ru:0 kk:0 en:0 other:5 | ru:0 kk:0 en:0 other:0 | | mms-lid-4017 r5 | ru:2 kk:2 en:86 other:2 | ru:1 kk:1 en:12 other:1 | ru:0 kk:0 en:1 other:0 | ru:0 kk:0 en:0 other:0 | | whisper full | ru:0 kk:0 en:13 other:2 | ru:4 kk:0 en:1 other:10 | ru:0 kk:3 en:0 other:12 | ru:0 kk:1 en:0 other:14 | | whisper r5 | ru:0 kk:0 en:15 other:0 | ru:8 kk:2 en:3 other:2 | ru:4 kk:8 en:2 other:1 | ru:0 kk:14 en:1 other:0 | #### E. In-domain raw argmax non-mn rate, % (no threshold) | config | n | raw non-mn 1-2s | raw non-mn 2-3s | raw non-mn 3-5s | raw non-mn 5-10s | raw non-mn 10-infs | |:------------------|-----:|------------------:|------------------:|------------------:|-------------------:|---------------------:| | ambernet full | 1000 | 25 | 7.5 | 6 | 2 | 1 | | ambernet r5 | 1000 | 11 | 3.5 | 3.5 | 1.5 | 1 | | speechbrain full | 1000 | 26 | 10.5 | 7 | 3 | 1 | | speechbrain r5 | 1000 | 10 | 4.5 | 5 | 1.5 | 1 | | mms-lid-126 full | 1000 | 32.5 | 10.5 | 7.5 | 3 | 0.5 | | mms-lid-126 r5 | 1000 | 18 | 7 | 5.5 | 2.5 | 0.5 | | mms-lid-256 full | 1000 | 36 | 9.5 | 7 | 2 | 0.5 | | mms-lid-256 r5 | 1000 | 21.5 | 6 | 5.5 | 2 | 0.5 | | mms-lid-4017 full | 1000 | 33 | 10.5 | 8 | 2.5 | 0.5 | | mms-lid-4017 r5 | 1000 | 20 | 7 | 6 | 2 | 0.5 | | whisper full | 50 | 100 | 100 | 100 | 100 | 100 | | whisper r5 | 50 | 100 | 100 | 100 | 100 | 100 | ### F. Speed and memory (CPU, one clip, median of 5 after warm-up; host busy, load1 8.9 to 12.8 from other bake-offs, so absolute times are pessimistic) | model | 5 s clip | 30 s clip | RAM (RSS after load / peak) | device | |---|---:|---:|---|---| | AmberNet ONNX (ORT CPU, default threads) | 101 ms | 953 ms | 197 / 365 MB | CPU | | SpeechBrain ECAPA (torch, 4 threads) | 515 ms | 2956 ms | 718 / 782 MB | CPU | | MMS-LID-126 (torch fp32, 4 threads) | 3911 ms | 25472 ms | 898 (mmap) / 4581 MB | CPU too slow for prod; GPU fp16 used for the eval | | MMS-LID-256 / 4017 | not timed, same 1B wav2vec2 architecture and 3.6 / 3.7 GB weights | | | | | Whisper turbo detect | not timed (30 s padded encoder; about 8 s per clip observed under load 30+) | | | | MMS GPU fp16: `torch.cuda.max_memory_allocated` peak 2.2 GB (126). nvidia-smi showed up to about 3.4 GB on the first run (caching allocator, 30 s clips), which exceeded the 2.5 GB target briefly; later runs used `expandable_segments` + `empty_cache` (about 2.6 GB). GPU was shared at 100% util by other jobs. ### G. Models, licenses, install | model | weights | license (as found) | size | |---|---|---|---| | AmberNet (`nvidia/langid_ambernet`, VoxLingua107, 107 langs) | `monsum-speech/models/ambernet-langid/` | model README: `license: other`, NVIDIA NGC terms of use. Not verified here; check commercial terms before relying on it | 29 M params | | SpeechBrain `speechbrain/lang-id-voxlingua107-ecapa` | HF | Apache-2.0 (model card) | 81 MB ckpt | | `facebook/mms-lid-126`, `-256`, `-4017` | HF | CC-BY-NC 4.0 (model cards) | about 3.6 to 3.7 GB each (1B params) | | Whisper large-v3-turbo (CT2) | local | MIT (upstream; not re-checked) | | Mongolian label per model: AmberNet/SpeechBrain/Whisper `mn`; MMS `mon` in all three (126, 256, 4017); `khk` absent there. Whisper turbo (base) labels Mongolian as Tibetan `bo` (15/15 FLEURS mn 5 s crops, p(mn)=0.01), which is a switchable language, so it would divert Mongolian to the wrong decode. Install: venvs under `/mnt/data01/Library/bakeoff-lid/` (`venv` py3.11, `venv12` py3.12 needed because monsum-speech uses PEP 695 syntax). `uv pip install numpy soundfile librosa onnxruntime pandas torch torchaudio speechbrain transformers huggingface_hub structlog starlette fastapi python-dotenv pydantic typer psutil tabulate faster-whisper`. MMS + SpeechBrain weights downloaded with `hf download` (about 8 min per MMS model on this link). Candidates skipped: none for download time. Searched quickly for newer credible LID with mn+kk (NVIDIA TitaNet-LID, Omnilingual LID): search found nothing confirming open weights with Mongolian and Kazakh, so not tested; I did not dig further. ### H. Caveats - Residual in-domain false-foreign at >= 5 s is mostly real foreign speech: 8 segments are called the same foreign language at p>=0.8 by AmberNet, SpeechBrain and MMS-126 together; 6 of the 8 are English spoken and transcribed in Cyrillic (e.g. "Окэй эйни квэйсшионс лийдэрс"). So the true Mongolian false-foreign floor at long durations is near 0, and the ~1% at 10+ s is mostly correct detection that the transcript script hides. At < 3 s the residual (2% to 5%) is real (e.g. "За нэг хоёрын гурав аа." read as `kk` by all three). - FLEURS mn here is `fleurs-local` read speech; in-domain is conversational/podcast. FLEURS crops include leading silence. - Whisper reference is a small subset; MMS uses fp16 GPU. ## Integrating a non-AmberNet winner No winner, so not needed. For reference: SpeechBrain ECAPA is Apache-2.0 and 81 MB, so it could be added behind `lid.py` using torch (already in the `ml` extra) with speechbrain as a new dep, or exported to ONNX (Fbank frontend + ECAPA; a few hours of work), but it gives no accuracy gain over AmberNet (r5 3 s recall ru 0.93 vs 0.97, zh equal, kk 0.68 vs 0.75 at 0.8) and is 5x slower. MMS is CC-BY-NC, 3.6 GB, 1B params, needs a GPU or ~4 s per 5 s on CPU, and also does not win: skip. ## Appendix: full per-language tables (decision rule, confusion, by model) ## ambernet (n clips: {'fleurs': 1993, 'indomain': 1000, 'req30': 108}) ### ambernet (all labels) FLEURS top-1 accuracy (raw argmax, no threshold) | lang | 1s | 2s | 3s | 5s | |:-------|:--------------|:--------------|:--------------|:--------------| | mn | 0.050 (n=100) | 0.600 (n=100) | 0.900 (n=100) | 0.990 (n=100) | | en | 0.050 (n=100) | 0.650 (n=100) | 0.840 (n=100) | 0.968 (n=94) | | ru | 0.150 (n=100) | 0.460 (n=100) | 0.690 (n=100) | 0.910 (n=100) | | zh | 0.020 (n=100) | 0.480 (n=100) | 0.750 (n=100) | 0.939 (n=99) | | kk | 0.050 (n=100) | 0.280 (n=100) | 0.640 (n=100) | 0.930 (n=100) | FLEURS mn raw errors (predicted label : count) | crop | mn-> | |:-------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | 1s | nn:33, haw:15, mi:9, jw:6, fo:6, da:3, th:2, en:2, war:2, hy:1, vi:1, fa:1, ja:1, ro:1, ht:1, kk:1, is:1, am:1, yi:1, eo:1, tt:1, ca:1, cy:1, bo:1, sv:1, sr:1 | | 2s | ka:3, nn:3, sr:3, haw:2, uz:2, mi:2, hi:2, en:2, ko:2, bn:1, ha:1, bo:1, tg:1, af:1, ja:1, fr:1, lb:1, fo:1, jw:1, ml:1, km:1, br:1, sq:1, sk:1, tt:1, ne:1, ar:1, kk:1 | | 3s | kk:1, mr:1, mg:1, ka:1, nn:1, ko:1, sk:1, uz:1, bs:1, sw:1 | | 5s | bs:1 | FLEURS decision rule (single clip, no speaker pooling) | crop | thr | mn false-foreign | recall en | recall ru | recall zh | recall kk | |:-------|------:|:-------------------|------------:|------------:|------------:|------------:| | 1s | 0.5 | 18/100 (18.0%) | 0.04 | 0.1 | 0.02 | 0.04 | | 1s | 0.7 | 10/100 (10.0%) | 0.03 | 0.06 | 0.02 | 0.04 | | 1s | 0.8 | 9/100 (9.0%) | 0.03 | 0.03 | 0.02 | 0.03 | | 1s | 0.9 | 6/100 (6.0%) | 0.03 | 0.03 | 0.02 | 0.03 | | 2s | 0.5 | 25/100 (25.0%) | 0.61 | 0.44 | 0.46 | 0.27 | | 2s | 0.7 | 13/100 (13.0%) | 0.47 | 0.38 | 0.39 | 0.22 | | 2s | 0.8 | 10/100 (10.0%) | 0.38 | 0.36 | 0.35 | 0.21 | | 2s | 0.9 | 8/100 (8.0%) | 0.29 | 0.31 | 0.35 | 0.2 | | 3s | 0.5 | 6/100 (6.0%) | 0.77 | 0.67 | 0.75 | 0.62 | | 3s | 0.7 | 3/100 (3.0%) | 0.66 | 0.6 | 0.75 | 0.59 | | 3s | 0.8 | 2/100 (2.0%) | 0.62 | 0.57 | 0.75 | 0.57 | | 3s | 0.9 | 1/100 (1.0%) | 0.5 | 0.49 | 0.75 | 0.52 | | 5s | 0.5 | 1/100 (1.0%) | 0.97 | 0.91 | 0.94 | 0.93 | | 5s | 0.7 | 0/100 (0.0%) | 0.88 | 0.82 | 0.92 | 0.89 | | 5s | 0.8 | 0/100 (0.0%) | 0.83 | 0.76 | 0.92 | 0.88 | | 5s | 0.9 | 0/100 (0.0%) | 0.7 | 0.64 | 0.92 | 0.85 | In-domain Mongolian false-foreign (by segment duration) | thr | 1-2s | 2-3s | 3-5s | 5-10s | 10-infs | all | |------:|:---------------|:-------------|:--------------|:-------------|:-------------|:---------------| | 0.5 | 28/200 (14.0%) | 9/200 (4.5%) | 11/200 (5.5%) | 1/200 (0.5%) | 1/200 (0.5%) | 50/1000 (5.0%) | | 0.7 | 14/200 (7.0%) | 4/200 (2.0%) | 6/200 (3.0%) | 0/200 (0.0%) | 1/200 (0.5%) | 25/1000 (2.5%) | | 0.8 | 9/200 (4.5%) | 4/200 (2.0%) | 4/200 (2.0%) | 0/200 (0.0%) | 1/200 (0.5%) | 18/1000 (1.8%) | | 0.9 | 5/200 (2.5%) | 3/200 (1.5%) | 1/200 (0.5%) | 0/200 (0.0%) | 1/200 (0.5%) | 10/1000 (1.0%) | raw argmax in-domain, top labels: {'mn': 917, 'sn': 4, 'en': 4, 'uz': 4, 'cy': 4, 'hu': 3} 30 s request clips (Mongolian) | thr | 30 s request false-foreign | |------:|:-----------------------------| | 0.5 | 0/108 | | 0.7 | 0/108 | | 0.8 | 0/108 | | 0.9 | 0/108 | ### ambernet (restricted to 5) FLEURS top-1 accuracy (raw argmax, no threshold) | lang | 1s | 2s | 3s | 5s | |:-------|:--------------|:--------------|:--------------|:--------------| | mn | 0.180 (n=100) | 0.800 (n=100) | 0.960 (n=100) | 1.000 (n=100) | | en | 0.610 (n=100) | 0.930 (n=100) | 0.990 (n=100) | 1.000 (n=94) | | ru | 0.400 (n=100) | 0.800 (n=100) | 0.970 (n=100) | 1.000 (n=100) | | zh | 0.150 (n=100) | 0.630 (n=100) | 0.760 (n=100) | 0.970 (n=99) | | kk | 0.120 (n=100) | 0.440 (n=100) | 0.800 (n=100) | 0.980 (n=100) | FLEURS mn raw errors (predicted label : count) | crop | mn-> | |:-------|:-----------------------| | 1s | en:62, ru:15, kk:5 | | 2s | kk:8, ru:6, en:5, zh:1 | | 3s | kk:3, en:1 | | 5s | none | FLEURS decision rule (single clip, no speaker pooling) | crop | thr | mn false-foreign | recall en | recall ru | recall zh | recall kk | |:-------|------:|:-------------------|------------:|------------:|------------:|------------:| | 1s | 0.5 | 50/100 (50.0%) | 0.48 | 0.36 | 0.13 | 0.12 | | 1s | 0.7 | 14/100 (14.0%) | 0.37 | 0.3 | 0.09 | 0.08 | | 1s | 0.8 | 9/100 (9.0%) | 0.32 | 0.28 | 0.09 | 0.07 | | 1s | 0.9 | 3/100 (3.0%) | 0.2 | 0.25 | 0.08 | 0.06 | | 2s | 0.5 | 13/100 (13.0%) | 0.93 | 0.79 | 0.61 | 0.44 | | 2s | 0.7 | 9/100 (9.0%) | 0.9 | 0.77 | 0.58 | 0.41 | | 2s | 0.8 | 7/100 (7.0%) | 0.86 | 0.76 | 0.56 | 0.38 | | 2s | 0.9 | 5/100 (5.0%) | 0.85 | 0.73 | 0.56 | 0.35 | | 3s | 0.5 | 3/100 (3.0%) | 0.99 | 0.97 | 0.76 | 0.77 | | 3s | 0.7 | 2/100 (2.0%) | 0.98 | 0.97 | 0.76 | 0.75 | | 3s | 0.8 | 1/100 (1.0%) | 0.98 | 0.97 | 0.76 | 0.75 | | 3s | 0.9 | 1/100 (1.0%) | 0.96 | 0.96 | 0.76 | 0.73 | | 5s | 0.5 | 0/100 (0.0%) | 1 | 1 | 0.97 | 0.98 | | 5s | 0.7 | 0/100 (0.0%) | 1 | 1 | 0.97 | 0.97 | | 5s | 0.8 | 0/100 (0.0%) | 1 | 1 | 0.97 | 0.94 | | 5s | 0.9 | 0/100 (0.0%) | 1 | 1 | 0.97 | 0.9 | In-domain Mongolian false-foreign (by segment duration) | thr | 1-2s | 2-3s | 3-5s | 5-10s | 10-infs | all | |------:|:--------------|:-------------|:-------------|:-------------|:-------------|:---------------| | 0.5 | 16/200 (8.0%) | 5/200 (2.5%) | 7/200 (3.5%) | 3/200 (1.5%) | 2/200 (1.0%) | 33/1000 (3.3%) | | 0.7 | 11/200 (5.5%) | 4/200 (2.0%) | 5/200 (2.5%) | 3/200 (1.5%) | 2/200 (1.0%) | 25/1000 (2.5%) | | 0.8 | 8/200 (4.0%) | 4/200 (2.0%) | 5/200 (2.5%) | 2/200 (1.0%) | 2/200 (1.0%) | 21/1000 (2.1%) | | 0.9 | 6/200 (3.0%) | 3/200 (1.5%) | 4/200 (2.0%) | 2/200 (1.0%) | 1/200 (0.5%) | 16/1000 (1.6%) | raw argmax in-domain, top labels: {'mn': 959, 'en': 19, 'kk': 13, 'ru': 5, 'zh': 4} 30 s request clips (Mongolian) | thr | 30 s request false-foreign | |------:|:-----------------------------| | 0.5 | 0/108 | | 0.7 | 0/108 | | 0.8 | 0/108 | | 0.9 | 0/108 | ## ambernet_r5 (n clips: {'fleurs': 1993, 'indomain': 1000, 'req30': 108}) ### ambernet_r5 (all labels) FLEURS top-1 accuracy (raw argmax, no threshold) | lang | 1s | 2s | 3s | 5s | |:-------|:--------------|:--------------|:--------------|:--------------| | mn | 0.180 (n=100) | 0.800 (n=100) | 0.960 (n=100) | 1.000 (n=100) | | en | 0.610 (n=100) | 0.930 (n=100) | 0.990 (n=100) | 1.000 (n=94) | | ru | 0.400 (n=100) | 0.800 (n=100) | 0.970 (n=100) | 1.000 (n=100) | | zh | 0.150 (n=100) | 0.630 (n=100) | 0.760 (n=100) | 0.970 (n=99) | | kk | 0.120 (n=100) | 0.440 (n=100) | 0.800 (n=100) | 0.980 (n=100) | FLEURS mn raw errors (predicted label : count) | crop | mn-> | |:-------|:-----------------------| | 1s | en:62, ru:15, kk:5 | | 2s | kk:8, ru:6, en:5, zh:1 | | 3s | kk:3, en:1 | | 5s | none | FLEURS decision rule (single clip, no speaker pooling) | crop | thr | mn false-foreign | recall en | recall ru | recall zh | recall kk | |:-------|------:|:-------------------|------------:|------------:|------------:|------------:| | 1s | 0.5 | 50/100 (50.0%) | 0.48 | 0.36 | 0.13 | 0.12 | | 1s | 0.7 | 14/100 (14.0%) | 0.37 | 0.3 | 0.09 | 0.08 | | 1s | 0.8 | 9/100 (9.0%) | 0.32 | 0.28 | 0.09 | 0.07 | | 1s | 0.9 | 3/100 (3.0%) | 0.2 | 0.25 | 0.08 | 0.06 | | 2s | 0.5 | 13/100 (13.0%) | 0.93 | 0.79 | 0.61 | 0.44 | | 2s | 0.7 | 9/100 (9.0%) | 0.9 | 0.77 | 0.58 | 0.41 | | 2s | 0.8 | 7/100 (7.0%) | 0.86 | 0.76 | 0.56 | 0.38 | | 2s | 0.9 | 5/100 (5.0%) | 0.85 | 0.73 | 0.56 | 0.35 | | 3s | 0.5 | 3/100 (3.0%) | 0.99 | 0.97 | 0.76 | 0.77 | | 3s | 0.7 | 2/100 (2.0%) | 0.98 | 0.97 | 0.76 | 0.75 | | 3s | 0.8 | 1/100 (1.0%) | 0.98 | 0.97 | 0.76 | 0.75 | | 3s | 0.9 | 1/100 (1.0%) | 0.96 | 0.96 | 0.76 | 0.73 | | 5s | 0.5 | 0/100 (0.0%) | 1 | 1 | 0.97 | 0.98 | | 5s | 0.7 | 0/100 (0.0%) | 1 | 1 | 0.97 | 0.97 | | 5s | 0.8 | 0/100 (0.0%) | 1 | 1 | 0.97 | 0.94 | | 5s | 0.9 | 0/100 (0.0%) | 1 | 1 | 0.97 | 0.9 | In-domain Mongolian false-foreign (by segment duration) | thr | 1-2s | 2-3s | 3-5s | 5-10s | 10-infs | all | |------:|:--------------|:-------------|:-------------|:-------------|:-------------|:---------------| | 0.5 | 16/200 (8.0%) | 5/200 (2.5%) | 7/200 (3.5%) | 3/200 (1.5%) | 2/200 (1.0%) | 33/1000 (3.3%) | | 0.7 | 11/200 (5.5%) | 4/200 (2.0%) | 5/200 (2.5%) | 3/200 (1.5%) | 2/200 (1.0%) | 25/1000 (2.5%) | | 0.8 | 8/200 (4.0%) | 4/200 (2.0%) | 5/200 (2.5%) | 2/200 (1.0%) | 2/200 (1.0%) | 21/1000 (2.1%) | | 0.9 | 6/200 (3.0%) | 3/200 (1.5%) | 4/200 (2.0%) | 2/200 (1.0%) | 1/200 (0.5%) | 16/1000 (1.6%) | raw argmax in-domain, top labels: {'mn': 959, 'en': 19, 'kk': 13, 'ru': 5, 'zh': 4} 30 s request clips (Mongolian) | thr | 30 s request false-foreign | |------:|:-----------------------------| | 0.5 | 0/108 | | 0.7 | 0/108 | | 0.8 | 0/108 | | 0.9 | 0/108 | ### ambernet_r5 (restricted to 5) FLEURS top-1 accuracy (raw argmax, no threshold) | lang | 1s | 2s | 3s | 5s | |:-------|:--------------|:--------------|:--------------|:--------------| | mn | 0.180 (n=100) | 0.800 (n=100) | 0.960 (n=100) | 1.000 (n=100) | | en | 0.610 (n=100) | 0.930 (n=100) | 0.990 (n=100) | 1.000 (n=94) | | ru | 0.400 (n=100) | 0.800 (n=100) | 0.970 (n=100) | 1.000 (n=100) | | zh | 0.150 (n=100) | 0.630 (n=100) | 0.760 (n=100) | 0.970 (n=99) | | kk | 0.120 (n=100) | 0.440 (n=100) | 0.800 (n=100) | 0.980 (n=100) | FLEURS mn raw errors (predicted label : count) | crop | mn-> | |:-------|:-----------------------| | 1s | en:62, ru:15, kk:5 | | 2s | kk:8, ru:6, en:5, zh:1 | | 3s | kk:3, en:1 | | 5s | none | FLEURS decision rule (single clip, no speaker pooling) | crop | thr | mn false-foreign | recall en | recall ru | recall zh | recall kk | |:-------|------:|:-------------------|------------:|------------:|------------:|------------:| | 1s | 0.5 | 50/100 (50.0%) | 0.48 | 0.36 | 0.13 | 0.12 | | 1s | 0.7 | 14/100 (14.0%) | 0.37 | 0.3 | 0.09 | 0.08 | | 1s | 0.8 | 9/100 (9.0%) | 0.32 | 0.28 | 0.09 | 0.07 | | 1s | 0.9 | 3/100 (3.0%) | 0.2 | 0.25 | 0.08 | 0.06 | | 2s | 0.5 | 13/100 (13.0%) | 0.93 | 0.79 | 0.61 | 0.44 | | 2s | 0.7 | 9/100 (9.0%) | 0.9 | 0.77 | 0.58 | 0.41 | | 2s | 0.8 | 7/100 (7.0%) | 0.86 | 0.76 | 0.56 | 0.38 | | 2s | 0.9 | 5/100 (5.0%) | 0.85 | 0.73 | 0.56 | 0.35 | | 3s | 0.5 | 3/100 (3.0%) | 0.99 | 0.97 | 0.76 | 0.77 | | 3s | 0.7 | 2/100 (2.0%) | 0.98 | 0.97 | 0.76 | 0.75 | | 3s | 0.8 | 1/100 (1.0%) | 0.98 | 0.97 | 0.76 | 0.75 | | 3s | 0.9 | 1/100 (1.0%) | 0.96 | 0.96 | 0.76 | 0.73 | | 5s | 0.5 | 0/100 (0.0%) | 1 | 1 | 0.97 | 0.98 | | 5s | 0.7 | 0/100 (0.0%) | 1 | 1 | 0.97 | 0.97 | | 5s | 0.8 | 0/100 (0.0%) | 1 | 1 | 0.97 | 0.94 | | 5s | 0.9 | 0/100 (0.0%) | 1 | 1 | 0.97 | 0.9 | In-domain Mongolian false-foreign (by segment duration) | thr | 1-2s | 2-3s | 3-5s | 5-10s | 10-infs | all | |------:|:--------------|:-------------|:-------------|:-------------|:-------------|:---------------| | 0.5 | 16/200 (8.0%) | 5/200 (2.5%) | 7/200 (3.5%) | 3/200 (1.5%) | 2/200 (1.0%) | 33/1000 (3.3%) | | 0.7 | 11/200 (5.5%) | 4/200 (2.0%) | 5/200 (2.5%) | 3/200 (1.5%) | 2/200 (1.0%) | 25/1000 (2.5%) | | 0.8 | 8/200 (4.0%) | 4/200 (2.0%) | 5/200 (2.5%) | 2/200 (1.0%) | 2/200 (1.0%) | 21/1000 (2.1%) | | 0.9 | 6/200 (3.0%) | 3/200 (1.5%) | 4/200 (2.0%) | 2/200 (1.0%) | 1/200 (0.5%) | 16/1000 (1.6%) | raw argmax in-domain, top labels: {'mn': 959, 'en': 19, 'kk': 13, 'ru': 5, 'zh': 4} 30 s request clips (Mongolian) | thr | 30 s request false-foreign | |------:|:-----------------------------| | 0.5 | 0/108 | | 0.7 | 0/108 | | 0.8 | 0/108 | | 0.9 | 0/108 | ## mms-lid-126 (n clips: {'fleurs': 1993, 'indomain': 1000, 'req30': 108}) ### mms-lid-126 (all labels) FLEURS top-1 accuracy (raw argmax, no threshold) | lang | 1s | 2s | 3s | 5s | |:-------|:--------------|:--------------|:--------------|:--------------| | mn | 0.050 (n=100) | 0.660 (n=100) | 0.970 (n=100) | 0.990 (n=100) | | en | 0.220 (n=100) | 0.900 (n=100) | 0.980 (n=100) | 1.000 (n=94) | | ru | 0.050 (n=100) | 0.540 (n=100) | 0.780 (n=100) | 0.960 (n=100) | | zh | 0.120 (n=100) | 0.550 (n=100) | 0.770 (n=100) | 0.970 (n=99) | | kk | 0.040 (n=100) | 0.330 (n=100) | 0.650 (n=100) | 0.930 (n=100) | FLEURS mn raw errors (predicted label : count) | crop | mn-> | |:-------|:-----------------------------------------------------------------------------------------------------------------------------------------| | 1s | cym:69, nno:17, dan:2, fao:1, hat:1, lat:1, kk:1, bre:1, haw:1, ru:1 | | 2s | cym:5, kk:4, tgk:3, fao:3, srp:2, lat:2, nno:2, orm:1, lav:1, bod:1, dan:1, mri:1, spa:1, ben:1, zh:1, kor:1, slv:1, zlm:1, heb:1, hun:1 | | 3s | kk:2, tgk:1 | | 5s | kir:1 | FLEURS decision rule (single clip, no speaker pooling) | crop | thr | mn false-foreign | recall en | recall ru | recall zh | recall kk | |:-------|------:|:-------------------|------------:|------------:|------------:|------------:| | 1s | 0.5 | 40/100 (40.0%) | 0.19 | 0.03 | 0.1 | 0.02 | | 1s | 0.7 | 26/100 (26.0%) | 0.17 | 0.02 | 0.09 | 0.02 | | 1s | 0.8 | 13/100 (13.0%) | 0.14 | 0.02 | 0.06 | 0.02 | | 1s | 0.9 | 4/100 (4.0%) | 0.12 | 0.02 | 0.05 | 0.02 | | 2s | 0.5 | 11/100 (11.0%) | 0.82 | 0.51 | 0.55 | 0.31 | | 2s | 0.7 | 4/100 (4.0%) | 0.78 | 0.46 | 0.49 | 0.26 | | 2s | 0.8 | 3/100 (3.0%) | 0.71 | 0.4 | 0.49 | 0.25 | | 2s | 0.9 | 2/100 (2.0%) | 0.61 | 0.33 | 0.41 | 0.2 | | 3s | 0.5 | 2/100 (2.0%) | 0.94 | 0.76 | 0.77 | 0.64 | | 3s | 0.7 | 2/100 (2.0%) | 0.93 | 0.74 | 0.76 | 0.63 | | 3s | 0.8 | 1/100 (1.0%) | 0.92 | 0.73 | 0.76 | 0.6 | | 3s | 0.9 | 1/100 (1.0%) | 0.88 | 0.67 | 0.76 | 0.51 | | 5s | 0.5 | 0/100 (0.0%) | 1 | 0.96 | 0.97 | 0.92 | | 5s | 0.7 | 0/100 (0.0%) | 0.99 | 0.95 | 0.96 | 0.91 | | 5s | 0.8 | 0/100 (0.0%) | 0.99 | 0.94 | 0.96 | 0.89 | | 5s | 0.9 | 0/100 (0.0%) | 0.96 | 0.92 | 0.96 | 0.87 | In-domain Mongolian false-foreign (by segment duration) | thr | 1-2s | 2-3s | 3-5s | 5-10s | 10-infs | all | |------:|:---------------|:--------------|:-------------|:-------------|:-------------|:---------------| | 0.5 | 20/200 (10.0%) | 10/200 (5.0%) | 8/200 (4.0%) | 6/200 (3.0%) | 1/200 (0.5%) | 45/1000 (4.5%) | | 0.7 | 11/200 (5.5%) | 8/200 (4.0%) | 4/200 (2.0%) | 4/200 (2.0%) | 0/200 (0.0%) | 27/1000 (2.7%) | | 0.8 | 8/200 (4.0%) | 7/200 (3.5%) | 4/200 (2.0%) | 4/200 (2.0%) | 0/200 (0.0%) | 23/1000 (2.3%) | | 0.9 | 5/200 (2.5%) | 5/200 (2.5%) | 3/200 (1.5%) | 2/200 (1.0%) | 0/200 (0.0%) | 15/1000 (1.5%) | raw argmax in-domain, top labels: {'mn': 892, 'bod': 10, 'cym': 9, 'kk': 7, 'nno': 4, 'sna': 3} 30 s request clips (Mongolian) | thr | 30 s request false-foreign | |------:|:-----------------------------| | 0.5 | 0/108 | | 0.7 | 0/108 | | 0.8 | 0/108 | | 0.9 | 0/108 | ### mms-lid-126 (restricted to 5) FLEURS top-1 accuracy (raw argmax, no threshold) | lang | 1s | 2s | 3s | 5s | |:-------|:--------------|:--------------|:--------------|:--------------| | mn | 0.080 (n=100) | 0.760 (n=100) | 0.980 (n=100) | 0.990 (n=100) | | en | 0.840 (n=100) | 0.970 (n=100) | 1.000 (n=100) | 1.000 (n=94) | | ru | 0.290 (n=100) | 0.750 (n=100) | 0.950 (n=100) | 1.000 (n=100) | | zh | 0.190 (n=100) | 0.610 (n=100) | 0.820 (n=100) | 0.970 (n=99) | | kk | 0.130 (n=100) | 0.490 (n=100) | 0.790 (n=100) | 0.940 (n=100) | FLEURS mn raw errors (predicted label : count) | crop | mn-> | |:-------|:------------------------| | 1s | en:88, kk:2, zh:1, ru:1 | | 2s | en:15, kk:8, zh:1 | | 3s | kk:2 | | 5s | kk:1 | FLEURS decision rule (single clip, no speaker pooling) | crop | thr | mn false-foreign | recall en | recall ru | recall zh | recall kk | |:-------|------:|:-------------------|------------:|------------:|------------:|------------:| | 1s | 0.5 | 88/100 (88.0%) | 0.76 | 0.25 | 0.18 | 0.09 | | 1s | 0.7 | 80/100 (80.0%) | 0.6 | 0.18 | 0.16 | 0.09 | | 1s | 0.8 | 63/100 (63.0%) | 0.46 | 0.17 | 0.13 | 0.07 | | 1s | 0.9 | 23/100 (23.0%) | 0.35 | 0.14 | 0.13 | 0.04 | | 2s | 0.5 | 20/100 (20.0%) | 0.97 | 0.74 | 0.61 | 0.45 | | 2s | 0.7 | 10/100 (10.0%) | 0.95 | 0.72 | 0.61 | 0.43 | | 2s | 0.8 | 7/100 (7.0%) | 0.94 | 0.72 | 0.6 | 0.4 | | 2s | 0.9 | 3/100 (3.0%) | 0.92 | 0.69 | 0.6 | 0.37 | | 3s | 0.5 | 2/100 (2.0%) | 1 | 0.95 | 0.81 | 0.79 | | 3s | 0.7 | 2/100 (2.0%) | 0.99 | 0.94 | 0.8 | 0.74 | | 3s | 0.8 | 2/100 (2.0%) | 0.99 | 0.93 | 0.8 | 0.72 | | 3s | 0.9 | 1/100 (1.0%) | 0.99 | 0.92 | 0.8 | 0.71 | | 5s | 0.5 | 1/100 (1.0%) | 1 | 1 | 0.97 | 0.94 | | 5s | 0.7 | 0/100 (0.0%) | 1 | 1 | 0.97 | 0.93 | | 5s | 0.8 | 0/100 (0.0%) | 1 | 1 | 0.97 | 0.92 | | 5s | 0.9 | 0/100 (0.0%) | 1 | 1 | 0.97 | 0.89 | In-domain Mongolian false-foreign (by segment duration) | thr | 1-2s | 2-3s | 3-5s | 5-10s | 10-infs | all | |------:|:---------------|:--------------|:-------------|:-------------|:-------------|:---------------| | 0.5 | 24/200 (12.0%) | 12/200 (6.0%) | 8/200 (4.0%) | 4/200 (2.0%) | 1/200 (0.5%) | 49/1000 (4.9%) | | 0.7 | 9/200 (4.5%) | 6/200 (3.0%) | 8/200 (4.0%) | 3/200 (1.5%) | 1/200 (0.5%) | 27/1000 (2.7%) | | 0.8 | 4/200 (2.0%) | 4/200 (2.0%) | 7/200 (3.5%) | 3/200 (1.5%) | 1/200 (0.5%) | 19/1000 (1.9%) | | 0.9 | 2/200 (1.0%) | 4/200 (2.0%) | 5/200 (2.5%) | 2/200 (1.0%) | 0/200 (0.0%) | 13/1000 (1.3%) | raw argmax in-domain, top labels: {'mn': 933, 'kk': 26, 'zh': 20, 'en': 14, 'ru': 7} 30 s request clips (Mongolian) | thr | 30 s request false-foreign | |------:|:-----------------------------| | 0.5 | 0/108 | | 0.7 | 0/108 | | 0.8 | 0/108 | | 0.9 | 0/108 | ## mms-lid-256 (n clips: {'fleurs': 1993, 'indomain': 1000, 'req30': 108}) ### mms-lid-256 (all labels) FLEURS top-1 accuracy (raw argmax, no threshold) | lang | 1s | 2s | 3s | 5s | |:-------|:--------------|:--------------|:--------------|:--------------| | mn | 0.050 (n=100) | 0.610 (n=100) | 0.940 (n=100) | 1.000 (n=100) | | en | 0.210 (n=100) | 0.890 (n=100) | 0.960 (n=100) | 1.000 (n=94) | | ru | 0.050 (n=100) | 0.450 (n=100) | 0.770 (n=100) | 0.960 (n=100) | | zh | 0.100 (n=100) | 0.520 (n=100) | 0.800 (n=100) | 0.970 (n=99) | | kk | 0.050 (n=100) | 0.310 (n=100) | 0.660 (n=100) | 0.920 (n=100) | FLEURS mn raw errors (predicted label : count) | crop | mn-> | |:-------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | 1s | cym:45, nno:41, fao:1, lao:1, orm:1, bod:1, tat:1, uig:1, yue:1, war:1, bul:1 | | 2s | nno:6, kk:5, cym:4, hye:2, bos:2, orm:1, fas:1, lat:1, aze:1, bod:1, kor:1, nob:1, ara:1, uig:1, hrv:1, spa:1, ben:1, ful:1, zh:1, haz:1, fao:1, jpn:1, slv:1, zlm:1, hun:1 | | 3s | lat:1, kk:1, yid:1, kat:1, hye:1, haz:1 | | 5s | none | FLEURS decision rule (single clip, no speaker pooling) | crop | thr | mn false-foreign | recall en | recall ru | recall zh | recall kk | |:-------|------:|:-------------------|------------:|------------:|------------:|------------:| | 1s | 0.5 | 23/100 (23.0%) | 0.19 | 0.03 | 0.09 | 0.03 | | 1s | 0.7 | 1/100 (1.0%) | 0.19 | 0.02 | 0.08 | 0.01 | | 1s | 0.8 | 1/100 (1.0%) | 0.18 | 0.01 | 0.08 | 0.01 | | 1s | 0.9 | 1/100 (1.0%) | 0.12 | 0.01 | 0.06 | 0.01 | | 2s | 0.5 | 8/100 (8.0%) | 0.83 | 0.44 | 0.49 | 0.28 | | 2s | 0.7 | 3/100 (3.0%) | 0.77 | 0.39 | 0.46 | 0.25 | | 2s | 0.8 | 2/100 (2.0%) | 0.72 | 0.34 | 0.46 | 0.24 | | 2s | 0.9 | 2/100 (2.0%) | 0.66 | 0.29 | 0.42 | 0.2 | | 3s | 0.5 | 4/100 (4.0%) | 0.95 | 0.77 | 0.8 | 0.65 | | 3s | 0.7 | 4/100 (4.0%) | 0.92 | 0.68 | 0.78 | 0.63 | | 3s | 0.8 | 4/100 (4.0%) | 0.91 | 0.63 | 0.77 | 0.59 | | 3s | 0.9 | 3/100 (3.0%) | 0.9 | 0.54 | 0.71 | 0.56 | | 5s | 0.5 | 0/100 (0.0%) | 1 | 0.96 | 0.97 | 0.92 | | 5s | 0.7 | 0/100 (0.0%) | 1 | 0.88 | 0.97 | 0.92 | | 5s | 0.8 | 0/100 (0.0%) | 0.98 | 0.88 | 0.96 | 0.92 | | 5s | 0.9 | 0/100 (0.0%) | 0.98 | 0.76 | 0.96 | 0.88 | In-domain Mongolian false-foreign (by segment duration) | thr | 1-2s | 2-3s | 3-5s | 5-10s | 10-infs | all | |------:|:---------------|:-------------|:-------------|:-------------|:-------------|:---------------| | 0.5 | 28/200 (14.0%) | 8/200 (4.0%) | 5/200 (2.5%) | 2/200 (1.0%) | 0/200 (0.0%) | 43/1000 (4.3%) | | 0.7 | 16/200 (8.0%) | 6/200 (3.0%) | 4/200 (2.0%) | 1/200 (0.5%) | 0/200 (0.0%) | 27/1000 (2.7%) | | 0.8 | 9/200 (4.5%) | 5/200 (2.5%) | 4/200 (2.0%) | 1/200 (0.5%) | 0/200 (0.0%) | 19/1000 (1.9%) | | 0.9 | 6/200 (3.0%) | 4/200 (2.0%) | 3/200 (1.5%) | 1/200 (0.5%) | 0/200 (0.0%) | 14/1000 (1.4%) | raw argmax in-domain, top labels: {'mn': 890, 'bod': 11, 'kk': 7, 'cym': 7, 'bre': 5, 'nno': 4} 30 s request clips (Mongolian) | thr | 30 s request false-foreign | |------:|:-----------------------------| | 0.5 | 0/108 | | 0.7 | 0/108 | | 0.8 | 0/108 | | 0.9 | 0/108 | ### mms-lid-256 (restricted to 5) FLEURS top-1 accuracy (raw argmax, no threshold) | lang | 1s | 2s | 3s | 5s | |:-------|:--------------|:--------------|:--------------|:--------------| | mn | 0.070 (n=100) | 0.690 (n=100) | 0.970 (n=100) | 1.000 (n=100) | | en | 0.800 (n=100) | 0.980 (n=100) | 1.000 (n=100) | 1.000 (n=94) | | ru | 0.240 (n=100) | 0.720 (n=100) | 0.960 (n=100) | 1.000 (n=100) | | zh | 0.200 (n=100) | 0.650 (n=100) | 0.840 (n=100) | 0.970 (n=99) | | kk | 0.120 (n=100) | 0.480 (n=100) | 0.840 (n=100) | 0.970 (n=100) | FLEURS mn raw errors (predicted label : count) | crop | mn-> | |:-------|:-------------------------| | 1s | en:87, kk:3, zh:2, ru:1 | | 2s | kk:14, en:13, zh:3, ru:1 | | 3s | kk:3 | | 5s | none | FLEURS decision rule (single clip, no speaker pooling) | crop | thr | mn false-foreign | recall en | recall ru | recall zh | recall kk | |:-------|------:|:-------------------|------------:|------------:|------------:|------------:| | 1s | 0.5 | 89/100 (89.0%) | 0.63 | 0.24 | 0.18 | 0.11 | | 1s | 0.7 | 59/100 (59.0%) | 0.5 | 0.19 | 0.14 | 0.09 | | 1s | 0.8 | 33/100 (33.0%) | 0.37 | 0.15 | 0.13 | 0.07 | | 1s | 0.9 | 6/100 (6.0%) | 0.25 | 0.09 | 0.12 | 0.06 | | 2s | 0.5 | 23/100 (23.0%) | 0.98 | 0.72 | 0.64 | 0.46 | | 2s | 0.7 | 15/100 (15.0%) | 0.95 | 0.68 | 0.61 | 0.44 | | 2s | 0.8 | 8/100 (8.0%) | 0.93 | 0.68 | 0.61 | 0.42 | | 2s | 0.9 | 5/100 (5.0%) | 0.92 | 0.65 | 0.58 | 0.41 | | 3s | 0.5 | 2/100 (2.0%) | 1 | 0.94 | 0.84 | 0.82 | | 3s | 0.7 | 2/100 (2.0%) | 0.98 | 0.94 | 0.84 | 0.8 | | 3s | 0.8 | 2/100 (2.0%) | 0.98 | 0.93 | 0.83 | 0.79 | | 3s | 0.9 | 1/100 (1.0%) | 0.98 | 0.93 | 0.82 | 0.75 | | 5s | 0.5 | 0/100 (0.0%) | 1 | 1 | 0.97 | 0.97 | | 5s | 0.7 | 0/100 (0.0%) | 1 | 1 | 0.97 | 0.96 | | 5s | 0.8 | 0/100 (0.0%) | 1 | 1 | 0.97 | 0.96 | | 5s | 0.9 | 0/100 (0.0%) | 1 | 1 | 0.97 | 0.96 | In-domain Mongolian false-foreign (by segment duration) | thr | 1-2s | 2-3s | 3-5s | 5-10s | 10-infs | all | |------:|:---------------|:-------------|:-------------|:-------------|:-------------|:---------------| | 0.5 | 24/200 (12.0%) | 9/200 (4.5%) | 9/200 (4.5%) | 4/200 (2.0%) | 1/200 (0.5%) | 47/1000 (4.7%) | | 0.7 | 13/200 (6.5%) | 7/200 (3.5%) | 7/200 (3.5%) | 2/200 (1.0%) | 1/200 (0.5%) | 30/1000 (3.0%) | | 0.8 | 10/200 (5.0%) | 6/200 (3.0%) | 7/200 (3.5%) | 2/200 (1.0%) | 1/200 (0.5%) | 26/1000 (2.6%) | | 0.9 | 6/200 (3.0%) | 4/200 (2.0%) | 5/200 (2.5%) | 2/200 (1.0%) | 1/200 (0.5%) | 18/1000 (1.8%) | raw argmax in-domain, top labels: {'mn': 929, 'kk': 35, 'en': 16, 'ru': 11, 'zh': 9} 30 s request clips (Mongolian) | thr | 30 s request false-foreign | |------:|:-----------------------------| | 0.5 | 0/108 | | 0.7 | 0/108 | | 0.8 | 0/108 | | 0.9 | 0/108 | ## mms-lid-4017 (n clips: {'fleurs': 1993, 'indomain': 1000, 'req30': 108}) ### mms-lid-4017 (all labels) FLEURS top-1 accuracy (raw argmax, no threshold) | lang | 1s | 2s | 3s | 5s | |:-------|:--------------|:--------------|:--------------|:--------------| | mn | 0.050 (n=100) | 0.680 (n=100) | 0.950 (n=100) | 1.000 (n=100) | | en | 0.240 (n=100) | 0.890 (n=100) | 0.970 (n=100) | 1.000 (n=94) | | ru | 0.090 (n=100) | 0.460 (n=100) | 0.750 (n=100) | 0.940 (n=100) | | zh | 0.120 (n=100) | 0.580 (n=100) | 0.780 (n=100) | 0.960 (n=99) | | kk | 0.010 (n=100) | 0.260 (n=100) | 0.630 (n=100) | 0.860 (n=100) | FLEURS mn raw errors (predicted label : count) | crop | mn-> | |:-------|:-----------------------------------------------------------------------------------------------------------------------------------| | 1s | nno:75, cym:11, fao:1, tha:1, aiw:1, wim:1, tuk:1, uig:1, yue:1, jav:1, ru:1 | | 2s | nno:9, mlg:2, sna:2, tgk:2, isl:2, cym:2, amh:1, hye:1, bre:1, vie:1, bod:1, fao:1, ara:1, bul:1, nob:1, slv:1, sun:1, kk:1, hrv:1 | | 3s | bre:1, yid:1, kat:1, hye:1, cym:1 | | 5s | none | FLEURS decision rule (single clip, no speaker pooling) | crop | thr | mn false-foreign | recall en | recall ru | recall zh | recall kk | |:-------|------:|:-------------------|------------:|------------:|------------:|------------:| | 1s | 0.5 | 3/100 (3.0%) | 0.19 | 0.04 | 0.1 | 0 | | 1s | 0.7 | 0/100 (0.0%) | 0.18 | 0.03 | 0.07 | 0 | | 1s | 0.8 | 0/100 (0.0%) | 0.16 | 0 | 0.06 | 0 | | 1s | 0.9 | 0/100 (0.0%) | 0.13 | 0 | 0.03 | 0 | | 2s | 0.5 | 3/100 (3.0%) | 0.79 | 0.43 | 0.54 | 0.24 | | 2s | 0.7 | 2/100 (2.0%) | 0.74 | 0.34 | 0.51 | 0.21 | | 2s | 0.8 | 2/100 (2.0%) | 0.68 | 0.34 | 0.51 | 0.2 | | 2s | 0.9 | 2/100 (2.0%) | 0.66 | 0.27 | 0.5 | 0.19 | | 3s | 0.5 | 3/100 (3.0%) | 0.97 | 0.75 | 0.77 | 0.57 | | 3s | 0.7 | 3/100 (3.0%) | 0.95 | 0.68 | 0.76 | 0.53 | | 3s | 0.8 | 2/100 (2.0%) | 0.93 | 0.65 | 0.76 | 0.51 | | 3s | 0.9 | 1/100 (1.0%) | 0.89 | 0.53 | 0.75 | 0.47 | | 5s | 0.5 | 0/100 (0.0%) | 0.99 | 0.94 | 0.95 | 0.84 | | 5s | 0.7 | 0/100 (0.0%) | 0.99 | 0.9 | 0.95 | 0.77 | | 5s | 0.8 | 0/100 (0.0%) | 0.98 | 0.88 | 0.95 | 0.76 | | 5s | 0.9 | 0/100 (0.0%) | 0.98 | 0.7 | 0.95 | 0.75 | In-domain Mongolian false-foreign (by segment duration) | thr | 1-2s | 2-3s | 3-5s | 5-10s | 10-infs | all | |------:|:---------------|:--------------|:-------------|:-------------|:-------------|:---------------| | 0.5 | 27/200 (13.5%) | 13/200 (6.5%) | 8/200 (4.0%) | 3/200 (1.5%) | 0/200 (0.0%) | 51/1000 (5.1%) | | 0.7 | 13/200 (6.5%) | 9/200 (4.5%) | 6/200 (3.0%) | 3/200 (1.5%) | 0/200 (0.0%) | 31/1000 (3.1%) | | 0.8 | 10/200 (5.0%) | 8/200 (4.0%) | 5/200 (2.5%) | 2/200 (1.0%) | 0/200 (0.0%) | 25/1000 (2.5%) | | 0.9 | 6/200 (3.0%) | 7/200 (3.5%) | 4/200 (2.0%) | 1/200 (0.5%) | 0/200 (0.0%) | 18/1000 (1.8%) | raw argmax in-domain, top labels: {'mn': 891, 'kk': 11, 'bod': 10, 'cym': 8, 'sna': 5, 'tat': 5} 30 s request clips (Mongolian) | thr | 30 s request false-foreign | |------:|:-----------------------------| | 0.5 | 0/108 | | 0.7 | 0/108 | | 0.8 | 0/108 | | 0.9 | 0/108 | ### mms-lid-4017 (restricted to 5) FLEURS top-1 accuracy (raw argmax, no threshold) | lang | 1s | 2s | 3s | 5s | |:-------|:--------------|:--------------|:--------------|:--------------| | mn | 0.080 (n=100) | 0.850 (n=100) | 0.990 (n=100) | 1.000 (n=100) | | en | 0.810 (n=100) | 0.990 (n=100) | 1.000 (n=100) | 1.000 (n=94) | | ru | 0.230 (n=100) | 0.760 (n=100) | 0.950 (n=100) | 1.000 (n=100) | | zh | 0.180 (n=100) | 0.640 (n=100) | 0.830 (n=100) | 0.970 (n=99) | | kk | 0.040 (n=100) | 0.370 (n=100) | 0.800 (n=100) | 0.940 (n=100) | FLEURS mn raw errors (predicted label : count) | crop | mn-> | |:-------|:------------------------| | 1s | en:86, zh:2, ru:2, kk:2 | | 2s | en:12, ru:1, zh:1, kk:1 | | 3s | en:1 | | 5s | none | FLEURS decision rule (single clip, no speaker pooling) | crop | thr | mn false-foreign | recall en | recall ru | recall zh | recall kk | |:-------|------:|:-------------------|------------:|------------:|------------:|------------:| | 1s | 0.5 | 66/100 (66.0%) | 0.69 | 0.22 | 0.16 | 0.04 | | 1s | 0.7 | 4/100 (4.0%) | 0.39 | 0.19 | 0.15 | 0.04 | | 1s | 0.8 | 4/100 (4.0%) | 0.29 | 0.18 | 0.12 | 0.02 | | 1s | 0.9 | 3/100 (3.0%) | 0.26 | 0.16 | 0.12 | 0.02 | | 2s | 0.5 | 11/100 (11.0%) | 0.96 | 0.74 | 0.64 | 0.35 | | 2s | 0.7 | 4/100 (4.0%) | 0.96 | 0.73 | 0.63 | 0.35 | | 2s | 0.8 | 2/100 (2.0%) | 0.96 | 0.73 | 0.61 | 0.34 | | 2s | 0.9 | 2/100 (2.0%) | 0.95 | 0.68 | 0.6 | 0.33 | | 3s | 0.5 | 1/100 (1.0%) | 1 | 0.95 | 0.81 | 0.77 | | 3s | 0.7 | 1/100 (1.0%) | 1 | 0.94 | 0.81 | 0.73 | | 3s | 0.8 | 0/100 (0.0%) | 1 | 0.93 | 0.81 | 0.72 | | 3s | 0.9 | 0/100 (0.0%) | 1 | 0.93 | 0.8 | 0.7 | | 5s | 0.5 | 0/100 (0.0%) | 1 | 1 | 0.97 | 0.93 | | 5s | 0.7 | 0/100 (0.0%) | 1 | 1 | 0.97 | 0.93 | | 5s | 0.8 | 0/100 (0.0%) | 1 | 1 | 0.97 | 0.91 | | 5s | 0.9 | 0/100 (0.0%) | 1 | 1 | 0.97 | 0.91 | In-domain Mongolian false-foreign (by segment duration) | thr | 1-2s | 2-3s | 3-5s | 5-10s | 10-infs | all | |------:|:---------------|:-------------|:--------------|:-------------|:-------------|:---------------| | 0.5 | 23/200 (11.5%) | 8/200 (4.0%) | 10/200 (5.0%) | 3/200 (1.5%) | 1/200 (0.5%) | 45/1000 (4.5%) | | 0.7 | 14/200 (7.0%) | 6/200 (3.0%) | 7/200 (3.5%) | 2/200 (1.0%) | 1/200 (0.5%) | 30/1000 (3.0%) | | 0.8 | 7/200 (3.5%) | 5/200 (2.5%) | 6/200 (3.0%) | 2/200 (1.0%) | 1/200 (0.5%) | 21/1000 (2.1%) | | 0.9 | 4/200 (2.0%) | 4/200 (2.0%) | 6/200 (3.0%) | 2/200 (1.0%) | 0/200 (0.0%) | 16/1000 (1.6%) | raw argmax in-domain, top labels: {'mn': 929, 'en': 29, 'kk': 20, 'ru': 14, 'zh': 8} 30 s request clips (Mongolian) | thr | 30 s request false-foreign | |------:|:-----------------------------| | 0.5 | 0/108 | | 0.7 | 0/108 | | 0.8 | 0/108 | | 0.9 | 0/108 | ## speechbrain (n clips: {'fleurs': 1993, 'indomain': 1000, 'req30': 108}) ### speechbrain (all labels) FLEURS top-1 accuracy (raw argmax, no threshold) | lang | 1s | 2s | 3s | 5s | |:-------|:--------------|:--------------|:--------------|:--------------| | mn | 0.080 (n=100) | 0.660 (n=100) | 0.970 (n=100) | 0.990 (n=100) | | en | 0.110 (n=100) | 0.730 (n=100) | 0.950 (n=100) | 0.989 (n=94) | | ru | 0.080 (n=100) | 0.340 (n=100) | 0.520 (n=100) | 0.600 (n=100) | | zh | 0.040 (n=100) | 0.400 (n=100) | 0.720 (n=100) | 0.929 (n=99) | | kk | 0.040 (n=100) | 0.320 (n=100) | 0.650 (n=100) | 0.900 (n=100) | FLEURS mn raw errors (predicted label : count) | crop | mn-> | |:-------|:-----------------------------------------------------------------------------------------------------------------------------------------------| | 1s | nn:66, fo:6, jw:4, cy:3, la:2, vi:1, ja:1, is:1, en:1, br:1, bo:1, kk:1, bg:1, so:1, mr:1, sr:1 | | 2s | no:5, nn:4, tg:3, hi:2, as:1, bo:1, yi:1, ht:1, uz:1, ha:1, ro:1, ka:1, tl:1, hu:1, fo:1, sd:1, hy:1, lo:1, iw:1, la:1, am:1, ja:1, sr:1, jw:1 | | 3s | no:1, kk:1, ko:1 | | 5s | kk:1 | FLEURS decision rule (single clip, no speaker pooling) | crop | thr | mn false-foreign | recall en | recall ru | recall zh | recall kk | |:-------|------:|:-------------------|------------:|------------:|------------:|------------:| | 1s | 0.5 | 3/100 (3.0%) | 0.07 | 0.05 | 0.03 | 0.04 | | 1s | 0.7 | 1/100 (1.0%) | 0.05 | 0.01 | 0.02 | 0.02 | | 1s | 0.8 | 1/100 (1.0%) | 0.03 | 0.01 | 0.01 | 0.02 | | 1s | 0.9 | 0/100 (0.0%) | 0.02 | 0 | 0.01 | 0.02 | | 2s | 0.5 | 7/100 (7.0%) | 0.62 | 0.33 | 0.39 | 0.29 | | 2s | 0.7 | 3/100 (3.0%) | 0.5 | 0.27 | 0.37 | 0.26 | | 2s | 0.8 | 2/100 (2.0%) | 0.43 | 0.2 | 0.37 | 0.2 | | 2s | 0.9 | 1/100 (1.0%) | 0.36 | 0.1 | 0.32 | 0.18 | | 3s | 0.5 | 3/100 (3.0%) | 0.89 | 0.49 | 0.72 | 0.57 | | 3s | 0.7 | 2/100 (2.0%) | 0.84 | 0.41 | 0.72 | 0.5 | | 3s | 0.8 | 1/100 (1.0%) | 0.79 | 0.32 | 0.7 | 0.47 | | 3s | 0.9 | 1/100 (1.0%) | 0.64 | 0.23 | 0.67 | 0.35 | | 5s | 0.5 | 1/100 (1.0%) | 0.99 | 0.58 | 0.93 | 0.89 | | 5s | 0.7 | 0/100 (0.0%) | 0.97 | 0.44 | 0.9 | 0.82 | | 5s | 0.8 | 0/100 (0.0%) | 0.96 | 0.35 | 0.89 | 0.78 | | 5s | 0.9 | 0/100 (0.0%) | 0.9 | 0.23 | 0.89 | 0.72 | In-domain Mongolian false-foreign (by segment duration) | thr | 1-2s | 2-3s | 3-5s | 5-10s | 10-infs | all | |------:|:---------------|:--------------|:-------------|:-------------|:-------------|:---------------| | 0.5 | 30/200 (15.0%) | 12/200 (6.0%) | 9/200 (4.5%) | 3/200 (1.5%) | 1/200 (0.5%) | 55/1000 (5.5%) | | 0.7 | 12/200 (6.0%) | 6/200 (3.0%) | 5/200 (2.5%) | 1/200 (0.5%) | 1/200 (0.5%) | 25/1000 (2.5%) | | 0.8 | 9/200 (4.5%) | 1/200 (0.5%) | 4/200 (2.0%) | 1/200 (0.5%) | 1/200 (0.5%) | 16/1000 (1.6%) | | 0.9 | 3/200 (1.5%) | 0/200 (0.0%) | 2/200 (1.0%) | 1/200 (0.5%) | 1/200 (0.5%) | 7/1000 (0.7%) | raw argmax in-domain, top labels: {'mn': 905, 'be': 6, 'cy': 4, 'en': 4, 'fa': 3, 'kk': 3} 30 s request clips (Mongolian) | thr | 30 s request false-foreign | |------:|:-----------------------------| | 0.5 | 0/108 | | 0.7 | 0/108 | | 0.8 | 0/108 | | 0.9 | 0/108 | ### speechbrain (restricted to 5) FLEURS top-1 accuracy (raw argmax, no threshold) | lang | 1s | 2s | 3s | 5s | |:-------|:--------------|:--------------|:--------------|:--------------| | mn | 0.280 (n=100) | 0.870 (n=100) | 0.990 (n=100) | 0.990 (n=100) | | en | 0.660 (n=100) | 0.960 (n=100) | 1.000 (n=100) | 1.000 (n=94) | | ru | 0.400 (n=100) | 0.780 (n=100) | 0.970 (n=100) | 1.000 (n=100) | | zh | 0.190 (n=100) | 0.640 (n=100) | 0.790 (n=100) | 0.960 (n=99) | | kk | 0.150 (n=100) | 0.510 (n=100) | 0.790 (n=100) | 0.940 (n=100) | FLEURS mn raw errors (predicted label : count) | crop | mn-> | |:-------|:------------------------| | 1s | en:63, kk:6, ru:2, zh:1 | | 2s | kk:7, en:6 | | 3s | kk:1 | | 5s | kk:1 | FLEURS decision rule (single clip, no speaker pooling) | crop | thr | mn false-foreign | recall en | recall ru | recall zh | recall kk | |:-------|------:|:-------------------|------------:|------------:|------------:|------------:| | 1s | 0.5 | 33/100 (33.0%) | 0.52 | 0.36 | 0.14 | 0.14 | | 1s | 0.7 | 11/100 (11.0%) | 0.4 | 0.34 | 0.12 | 0.12 | | 1s | 0.8 | 5/100 (5.0%) | 0.38 | 0.32 | 0.11 | 0.1 | | 1s | 0.9 | 2/100 (2.0%) | 0.31 | 0.28 | 0.1 | 0.08 | | 2s | 0.5 | 10/100 (10.0%) | 0.96 | 0.78 | 0.64 | 0.48 | | 2s | 0.7 | 7/100 (7.0%) | 0.95 | 0.76 | 0.63 | 0.46 | | 2s | 0.8 | 4/100 (4.0%) | 0.94 | 0.74 | 0.62 | 0.42 | | 2s | 0.9 | 2/100 (2.0%) | 0.92 | 0.7 | 0.58 | 0.4 | | 3s | 0.5 | 1/100 (1.0%) | 1 | 0.97 | 0.78 | 0.76 | | 3s | 0.7 | 1/100 (1.0%) | 1 | 0.96 | 0.77 | 0.74 | | 3s | 0.8 | 1/100 (1.0%) | 1 | 0.93 | 0.76 | 0.68 | | 3s | 0.9 | 0/100 (0.0%) | 1 | 0.93 | 0.74 | 0.64 | | 5s | 0.5 | 1/100 (1.0%) | 1 | 1 | 0.96 | 0.94 | | 5s | 0.7 | 1/100 (1.0%) | 1 | 1 | 0.95 | 0.91 | | 5s | 0.8 | 0/100 (0.0%) | 1 | 1 | 0.95 | 0.9 | | 5s | 0.9 | 0/100 (0.0%) | 1 | 1 | 0.93 | 0.88 | In-domain Mongolian false-foreign (by segment duration) | thr | 1-2s | 2-3s | 3-5s | 5-10s | 10-infs | all | |------:|:--------------|:-------------|:-------------|:-------------|:-------------|:---------------| | 0.5 | 15/200 (7.5%) | 8/200 (4.0%) | 9/200 (4.5%) | 3/200 (1.5%) | 2/200 (1.0%) | 37/1000 (3.7%) | | 0.7 | 9/200 (4.5%) | 5/200 (2.5%) | 6/200 (3.0%) | 3/200 (1.5%) | 2/200 (1.0%) | 25/1000 (2.5%) | | 0.8 | 8/200 (4.0%) | 5/200 (2.5%) | 4/200 (2.0%) | 2/200 (1.0%) | 2/200 (1.0%) | 21/1000 (2.1%) | | 0.9 | 3/200 (1.5%) | 4/200 (2.0%) | 4/200 (2.0%) | 2/200 (1.0%) | 2/200 (1.0%) | 15/1000 (1.5%) | raw argmax in-domain, top labels: {'mn': 956, 'kk': 19, 'en': 14, 'ru': 10, 'zh': 1} 30 s request clips (Mongolian) | thr | 30 s request false-foreign | |------:|:-----------------------------| | 0.5 | 0/108 | | 0.7 | 0/108 | | 0.8 | 0/108 | | 0.9 | 0/108 | ## whisper (n clips: {'fleurs': 298, 'indomain': 50, 'req30': 0}) ### whisper (all labels) FLEURS top-1 accuracy (raw argmax, no threshold) | lang | 1s | 2s | 3s | 5s | |:-------|:-------------|:-------------|:-------------|:-------------| | mn | 0.000 (n=15) | 0.000 (n=15) | 0.000 (n=15) | 0.000 (n=15) | | en | 1.000 (n=15) | 1.000 (n=15) | 1.000 (n=15) | 1.000 (n=13) | | ru | 0.333 (n=15) | 0.533 (n=15) | 0.867 (n=15) | 1.000 (n=15) | | zh | 0.267 (n=15) | 0.600 (n=15) | 0.733 (n=15) | 1.000 (n=15) | | kk | 0.000 (n=15) | 0.200 (n=15) | 0.600 (n=15) | 0.933 (n=15) | FLEURS mn raw errors (predicted label : count) | crop | mn-> | |:-------|:-----------------------------------------------------------------| | 1s | en:13, es:1, is:1 | | 2s | ru:4, ko:2, it:1, en:1, zh:1, hy:1, es:1, bo:1, tr:1, is:1, ar:1 | | 3s | bo:5, kk:3, ko:3, hy:1, ar:1, mk:1, bg:1 | | 5s | bo:13, hy:1, kk:1 | FLEURS decision rule (single clip, no speaker pooling) | crop | thr | mn false-foreign | recall en | recall ru | recall zh | recall kk | |:-------|------:|:-------------------|------------:|------------:|------------:|------------:| | 1s | 0.5 | 0/15 (0.0%) | 0.53 | 0.33 | 0.13 | 0 | | 1s | 0.7 | 0/15 (0.0%) | 0.4 | 0.2 | 0.13 | 0 | | 1s | 0.8 | 0/15 (0.0%) | 0.33 | 0.2 | 0.13 | 0 | | 1s | 0.9 | 0/15 (0.0%) | 0.33 | 0.13 | 0.13 | 0 | | 2s | 0.5 | 5/15 (33.3%) | 1 | 0.47 | 0.6 | 0.2 | | 2s | 0.7 | 2/15 (13.3%) | 1 | 0.47 | 0.6 | 0.2 | | 2s | 0.8 | 1/15 (6.7%) | 1 | 0.47 | 0.6 | 0.07 | | 2s | 0.9 | 1/15 (6.7%) | 1 | 0.47 | 0.53 | 0.07 | | 3s | 0.5 | 9/15 (60.0%) | 1 | 0.87 | 0.73 | 0.53 | | 3s | 0.7 | 4/15 (26.7%) | 1 | 0.87 | 0.73 | 0.47 | | 3s | 0.8 | 3/15 (20.0%) | 1 | 0.87 | 0.73 | 0.47 | | 3s | 0.9 | 0/15 (0.0%) | 1 | 0.87 | 0.73 | 0.47 | | 5s | 0.5 | 7/15 (46.7%) | 1 | 1 | 1 | 0.93 | | 5s | 0.7 | 1/15 (6.7%) | 1 | 1 | 1 | 0.93 | | 5s | 0.8 | 1/15 (6.7%) | 1 | 1 | 1 | 0.93 | | 5s | 0.9 | 0/15 (0.0%) | 1 | 1 | 1 | 0.93 | In-domain Mongolian false-foreign (by segment duration) | thr | 1-2s | 2-3s | 3-5s | 5-10s | 10-infs | all | |------:|:-------------|:-------------|:-------------|:------------|:------------|:-------------| | 0.5 | 3/10 (30.0%) | 1/10 (10.0%) | 3/10 (30.0%) | 0/10 (0.0%) | 0/10 (0.0%) | 7/50 (14.0%) | | 0.7 | 0/10 (0.0%) | 0/10 (0.0%) | 1/10 (10.0%) | 0/10 (0.0%) | 0/10 (0.0%) | 1/50 (2.0%) | | 0.8 | 0/10 (0.0%) | 0/10 (0.0%) | 1/10 (10.0%) | 0/10 (0.0%) | 0/10 (0.0%) | 1/50 (2.0%) | | 0.9 | 0/10 (0.0%) | 0/10 (0.0%) | 1/10 (10.0%) | 0/10 (0.0%) | 0/10 (0.0%) | 1/50 (2.0%) | raw argmax in-domain, top labels: {'cy': 11, 'en': 8, 'ru': 7, 'kk': 5, 'bo': 4, 'ko': 3} 30 s request clips (Mongolian) | thr | 30 s request false-foreign | |------:|:-----------------------------| | 0.5 | 0/0 | | 0.7 | 0/0 | | 0.8 | 0/0 | | 0.9 | 0/0 | ### whisper (restricted to 5) FLEURS top-1 accuracy (raw argmax, no threshold) | lang | 1s | 2s | 3s | 5s | |:-------|:-------------|:-------------|:-------------|:-------------| | mn | 0.000 (n=15) | 0.000 (n=15) | 0.000 (n=15) | 0.000 (n=15) | | en | 1.000 (n=15) | 1.000 (n=15) | 1.000 (n=15) | 1.000 (n=13) | | ru | 0.333 (n=15) | 0.600 (n=15) | 0.933 (n=15) | 1.000 (n=15) | | zh | 0.267 (n=15) | 0.600 (n=15) | 0.867 (n=15) | 1.000 (n=15) | | kk | 0.000 (n=15) | 0.267 (n=15) | 0.600 (n=15) | 0.933 (n=15) | FLEURS mn raw errors (predicted label : count) | crop | mn-> | |:-------|:-----------------------| | 1s | en:15 | | 2s | ru:8, en:3, zh:2, kk:2 | | 3s | kk:8, ru:4, en:2, zh:1 | | 5s | kk:14, en:1 | FLEURS decision rule (single clip, no speaker pooling) | crop | thr | mn false-foreign | recall en | recall ru | recall zh | recall kk | |:-------|------:|:-------------------|------------:|------------:|------------:|------------:| | 1s | 0.5 | 15/15 (100.0%) | 1 | 0.33 | 0.27 | 0 | | 1s | 0.7 | 9/15 (60.0%) | 0.93 | 0.33 | 0.2 | 0 | | 1s | 0.8 | 1/15 (6.7%) | 0.67 | 0.33 | 0.13 | 0 | | 1s | 0.9 | 0/15 (0.0%) | 0.47 | 0.27 | 0.13 | 0 | | 2s | 0.5 | 11/15 (73.3%) | 1 | 0.6 | 0.6 | 0.27 | | 2s | 0.7 | 7/15 (46.7%) | 1 | 0.6 | 0.6 | 0.2 | | 2s | 0.8 | 4/15 (26.7%) | 1 | 0.6 | 0.6 | 0.13 | | 2s | 0.9 | 2/15 (13.3%) | 1 | 0.53 | 0.53 | 0.07 | | 3s | 0.5 | 11/15 (73.3%) | 1 | 0.93 | 0.8 | 0.6 | | 3s | 0.7 | 8/15 (53.3%) | 1 | 0.93 | 0.73 | 0.53 | | 3s | 0.8 | 6/15 (40.0%) | 1 | 0.93 | 0.73 | 0.47 | | 3s | 0.9 | 5/15 (33.3%) | 1 | 0.93 | 0.73 | 0.47 | | 5s | 0.5 | 14/15 (93.3%) | 1 | 1 | 1 | 0.93 | | 5s | 0.7 | 13/15 (86.7%) | 1 | 1 | 1 | 0.93 | | 5s | 0.8 | 9/15 (60.0%) | 1 | 1 | 1 | 0.93 | | 5s | 0.9 | 5/15 (33.3%) | 1 | 1 | 1 | 0.93 | In-domain Mongolian false-foreign (by segment duration) | thr | 1-2s | 2-3s | 3-5s | 5-10s | 10-infs | all | |------:|:-------------|:-------------|:-------------|:-------------|:-------------|:--------------| | 0.5 | 6/10 (60.0%) | 7/10 (70.0%) | 9/10 (90.0%) | 9/10 (90.0%) | 5/10 (50.0%) | 36/50 (72.0%) | | 0.7 | 3/10 (30.0%) | 2/10 (20.0%) | 6/10 (60.0%) | 4/10 (40.0%) | 2/10 (20.0%) | 17/50 (34.0%) | | 0.8 | 2/10 (20.0%) | 2/10 (20.0%) | 6/10 (60.0%) | 1/10 (10.0%) | 1/10 (10.0%) | 12/50 (24.0%) | | 0.9 | 1/10 (10.0%) | 0/10 (0.0%) | 5/10 (50.0%) | 1/10 (10.0%) | 1/10 (10.0%) | 8/50 (16.0%) | raw argmax in-domain, top labels: {'en': 23, 'kk': 14, 'ru': 10, 'zh': 3} 30 s request clips (Mongolian) | thr | 30 s request false-foreign | |------:|:-----------------------------| | 0.5 | 0/0 | | 0.7 | 0/0 | | 0.8 | 0/0 | | 0.9 | 0/0 |