# Diarization bake-off (ASR words fixed, diarization swapped)

8 Mongolian eval files (31-94 min, 2-40 speakers). Baseline ASR = `hyp/lidfix` words. Candidates' turns re-attributed with
`monsum_speech.adapters.align.split_segments_by_speaker(max_gap=0.1)` + `drop_unattributed`, scored with `eval_speech.py score-turns`.

| model | cpWER | WER | mean abs spk err | ref>hyp speakers per file | diar wall | peak GPU MiB |
|---|---|---|---|---|---|---|
| pyannote community-1 (lidfix, full request) | 0.298 | 0.214 | 3.38 | 2>3, 4>4, 40>21, 8>7, 24>20, 10>10, 12>12, 6>8 | 1514 s | 4286 |
| nemotron3 | 0.399 | 0.236 | 7.00 | 2>2, 4>3, 40>8, 8>7, 24>8, 10>8, 12>8, 6>6 | 76 s | 3686 |
| diarizen | 0.282 | 0.214 | 3.12 | 2>4, 4>4, 40>19, 8>8, 24>23, 10>10, 12>12, 6>7 | 2070 s | 4074 |

- pyannote row = the lidfix response as served: wall and GPU are the **whole request** (ASR + diarization), not comparable to the other two rows. Its speaker labels come from the service's own attribution.
- GPU peak = nvidia-smi sampled every 0.5 s, increment over the pre-call baseline (weights already resident). Caching allocator keeps blocks, so later files in the same process read low (diarizen showed 0 on most); the max is the number reported.
- Nemotron3 is capped at 8 speakers: files with 40, 24, 12 and 10 speakers cannot score above 8 (cpWER 0.79 / 0.43 / 0.51 / 0.57 on those). Mean speaker error 7.0 is mostly this cap.
- DiariZen vs pyannote: cpWER 0.282 vs 0.298, WER equal (0.214). Lower cpWER on 7 of 8 files (only the 40-speaker file is worse: 0.455 vs 0.446). Over-splits the 2-speaker file (2>4); 40>19 is still under-counted.

## Method note
`split_segments_by_speaker` only reads each input segment's words and language, so the cached (already speaker-split) response segments are fed as-is with `speaker` cleared; no regrouping needed. `drop_unattributed_spans` (pre-ASR span filter in the pipeline) cannot be replayed with fixed ASR and is skipped, so a candidate's gain/loss from not decoding unattributed spans is not measured.

## Install notes
- Nemotron3: py3.12 venv, torch 2.15 (cu13 wheels, driver 615 ok), numpy 2.5.4, lightning 2.4.0, transformers 5.22.1. **PyPI `nemo-toolkit` 3.0.0 cannot load the checkpoint** (`self_attention_model='rope' is not supported`); needs NeMo Speech git main (3.1.0+e5d79d5a4, lhotse 2.0.0a6). Offline profile from the card (spkcache 264, fifo 40, chunk 340, right ctx 40, update 300), model 99 M params, ~13 s for a 94 min file.
- DiariZen (BUT-FIT/diarizen-wavlm-large-s80-md-v2): py3.10, **hard pins torch 2.1.1+cu121, numpy 1.26.4, bundled pyannote-audio fork (reports 3.1.1)**, speechbrain 1.1.1. Conflicts with monsum-speech's pyannote 4.0.7 / torch 2.8: cannot share a venv, would need a separate worker process/service. Config batch 32 OOMs on 12 GB (`MemoryError` in Inference.infer); segmentation batch 8 worked. Also pulls `pyannote/wespeaker-voxceleb-resnet34-LM` at load. Speed ~14x realtime (about 2070 s for 8 files).
- Everything lives in `/mnt/data01/Library/bakeoff-diar/` (venvs, hf, turns, run scripts in `run/`, logs).