# Speaker-embedding bake-off (ASR words fixed, segmentation fixed, embedding + clustering swapped)

Same 8 Mongolian eval files and the same scorer as `bakeoff.md`: `hyp/lidfix` ASR words kept, candidate turns re-attributed with
`split_segments_by_speaker(max_gap=0.1)` + `drop_unattributed`, scored by `eval_speech.py score-turns` (cpWER pooled over files). No oracle speaker count.
Overlapping `speaker_diarization` track (service default, `DIARIZATION_EXCLUSIVE` off). Nothing in `monsum-speech` was edited.

## Method
- One pass of community-1 segmentation per file, cached. Embeddings for every (10 s chunk, local speaker) pair cached per model. Clustering + `reconstruct` + `to_annotation` replay `SpeakerDiarization.apply` from the clustering call on, so only embedding and clustering differ between rows.
- Control A = checkpoint VBx (PLDA, threshold 0.6, Fa 0.07, Fb 0.8) on native embeddings. It reproduces `pyannote_metrics.json` exactly (cpWER 0.2979, WER 0.2137, spk err 3.375, same ref>hyp counts).
- AHC = pyannote `AgglomerativeClustering`, cosine, centroid linkage, `min_cluster_size=12` (pyannote 3.1 default), threshold swept. `min_cluster_size=1` was tried on r34: it shatters files into 30 to 330 speakers (scorer crashes beyond 62 hyp speakers), so it is dropped.
- **Two embedding front-ends, this matters.** `native` = pyannote's own ResNet34 with mask-weighted statistics pooling (what the service does). `crop` = keep only the samples the segmentation marks as that speaker, concatenate, tile-pad to the next whole second (to batch equal lengths), embed that waveform plainly. The third-party models (WeSpeaker SimAM, ReDimNet2) have no weighted-pooling hook, so they can only use `crop`. To keep the model effect separate from the front-end effect, community-1's own ResNet34 weights were also run through `crop` (`r34 crop`).
- Inactive speakers (empty mask) return NaN in `crop` runs; pyannote discards them downstream. The r34 crop run embedded them (wasted time, same output).

## Results (best global threshold = tuned on test; LOO = threshold picked on 7 files, scored on the 8th)

| model | clustering | thr | cpWER | LOO cpWER | WER | mean abs spk err | ref>hyp speakers per file | dim | license |
|---|---|---|---|---|---|---|---|---|---|
| A. community-1 ResNet34 native (baseline) | VBx/PLDA | 0.6 (ckpt) | 0.298 | n/a (no tuning) | 0.214 | 3.38 | 2>3, 4>4, 40>21, 8>7, 24>20, 10>10, 12>12, 6>8 | 256 | pyannote community-1 CC-BY-4.0 (WeSpeaker ResNet34: CC-BY-4.0 / Apache-2.0 code) |
| B. r34 native | AHC | 0.70 | 0.314 | 0.510 | 0.214 | 3.88 | 2>4, 4>6, 40>22, 8>7, 24>19, 10>10, 12>13, 6>8 | 256 | as A |
| B2. r34 crop | AHC | 0.70 | 0.267 | 0.270 | 0.214 | 3.12 | 2>2, 4>3, 40>26, 8>5, 24>20, 10>8, 12>11, 6>6 | 256 | as A |
| B3. r34 crop | VBx/PLDA (ckpt thr 0.6, untuned) | 0.6 | 0.268 | n/a | 0.214 | 4.12 | 2>2, 4>3, 40>17, 8>6, 24>20, 10>8, 12>11, 6>6 | 256 | as A |
| C1. WeSpeaker SimAM-ResNet34, VoxBlink2 | AHC | 0.70 | 0.278 | 0.483 | 0.214 | 2.50 | 2>2, 4>3, 40>28, 8>6, 24>20, 10>10, 12>11, 6>6 | 256 | WeSpeaker Apache-2.0; VoxBlink2 data CC-BY-NC |
| C2. WeSpeaker SimAM-ResNet100, VoxBlink2 | AHC | 0.725 | 0.266 | 0.432 | 0.214 | 3.00 | 2>2, 4>3, 40>27, 8>6, 24>19, 10>8, 12>11, 6>6 | 256 | as C1 |
| D. ReDimNet2-B6 (WeSpeaker, Vox2 + LM) | AHC | 0.675 | 0.268 | 0.272 | 0.214 | 3.12 | 2>2, 4>3, 40>25, 8>6, 24>19, 10>9, 12>11, 6>6 | 192 | WeSpeaker Apache-2.0; VoxCeleb data research-only |

Per-file cpWER (files in the order of the ref>hyp column, 2/4/40/8/24/10/12/6 speakers):

| model | 2 | 4 | 40 | 8 | 24 | 10 | 12 | 6 |
|---|---|---|---|---|---|---|---|---|
| A | 0.214 | 0.229 | 0.446 | 0.390 | 0.225 | 0.440 | 0.279 | 0.162 |
| B | 0.224 | 0.237 | 0.511 | 0.401 | 0.229 | 0.458 | 0.289 | 0.175 |
| B2 | 0.203 | 0.224 | 0.377 | 0.370 | 0.184 | 0.395 | 0.221 | 0.158 |
| B3 | 0.201 | 0.223 | 0.396 | 0.366 | 0.184 | 0.393 | 0.223 | 0.159 |
| C1 | 0.201 | 0.223 | 0.367 | 0.368 | 0.188 | 0.452 | 0.221 | 0.156 |
| C2 | 0.201 | 0.224 | 0.368 | 0.371 | 0.189 | 0.393 | 0.221 | 0.159 |
| D | 0.201 | 0.224 | 0.376 | 0.364 | 0.190 | 0.404 | 0.221 | 0.155 |

Threshold sweep, cpWER (AHC, mcs 12). Note the cliff at 0.75 for every model: the 10-speaker file collapses to 2 to 4 speakers there, so any LOO pick that lands on 0.75 is punished.

| thr | 0.5 | 0.6 | 0.65 | 0.675 | 0.7 | 0.725 | 0.75 | 0.8 | 0.9 |
|---|---|---|---|---|---|---|---|---|---|
| r34 native | 0.397 | 0.339 | 0.315 | 0.319 | 0.314 | 0.317 | 0.504 | 0.503 | 0.855 |
| r34 crop | 0.341 | 0.275 | 0.271 | 0.270 | 0.267 | 0.269 | 0.491 | 0.490 | 0.765 |
| SimAM-R34 | 0.329 | 0.289 | 0.285 | 0.278 | 0.278 | 0.292 | 0.483 | 0.554 | 0.997 |
| SimAM-R100 | 0.319 | 0.287 | 0.283 | 0.279 | 0.267 | 0.266 | 0.431 | 0.536 | 0.856 |
| ReDimNet2-B6 | 0.298 | 0.273 | 0.269 | 0.268 | 0.269 | 0.269 | 0.487 | 0.509 | 0.832 |

ReDimNet2 also: thr 0.4 = 0.470. r34 crop + VBx: thr 0.5 = 0.279, 0.6 = 0.268, 0.7 = 0.268.
Full sweep dumps: `/mnt/data01/Library/bakeoff-diar/embed/report.txt`, scores in `.../embed/scores/`, turns in `.../embed/turns/`.

## Cost (file 01a10a55, 31.5 min, 8 speakers, embedding only; clustering is 0.5 s AHC / 4 s VBx per file)
GPU shared with other users during all runs, so wall times are noisy (r34 native measured 43 s in one run and 82 s in another). Peak = nvidia-smi 0.5 s sampling of this process only, increment over the loaded-weights baseline.

| model | emb wall | peak VRAM (increment / total) |
|---|---|---|
| r34 native, batch 32 | 43 to 82 s | 2.0 / 2.2 GB |
| r34 crop | 29 s | 2.1 / 2.3 GB |
| SimAM-R34 crop | 88 s | 5.8 / 6.1 GB |
| SimAM-R100 crop | 248 s | 5.8 / 6.2 GB |
| ReDimNet2-B6 crop (sub-batch 4) | 197 s | 5.7 / 5.9 GB |

Sub-batch 16 for SimAM (and the 128-wave pipeline batch) is what drives VRAM to about 6 GB; sub-batch 4 to 8 would fit well under. Whole-set embedding wall (8 files, contended, mixed code versions, not comparable): r34 2422 s, r34 crop 2957 s, SimAM-R34 2734 s, SimAM-R100 6370 s, ReDimNet2 3065 s. Segmentation is shared, 2 to 12 s per file.

## Findings
1. **The embedding model barely matters.** r34 crop 0.267, SimAM-R100 0.266, ReDimNet2 0.268, SimAM-R34 0.278, all best-case on test and within noise of each other (8 files). The 256-d ResNet34 weights already in the checkpoint do as well as VoxBlink2-pretrained SimAM-ResNet100 and ReDimNet2-B6.
2. **The gain over baseline (0.298 to about 0.267, 10%) comes from the extraction front-end, not the model.** Same ResNet34 weights, crop-and-concatenate instead of mask-weighted pooling: 0.314 to 0.267 under AHC, and 0.298 to 0.268 under the checkpoint's own VBx with the untouched threshold 0.6 (B3, no tuning). Cause not isolated (it could be the plain pooling, the 1 s tile-padding, or both); worth an ablation before shipping.
3. **Speaker undercount on big panels is only partly fixed.** 40 speakers: 21 to 25-28 hyp with AHC (17 with crop + VBx); 24 speakers: 20 to 19-20. Better cpWER on the 40-speaker file (0.446 to 0.367-0.377) but still far from 40. Mean abs count error 3.38 to 2.5 (SimAM-R34) or 3.1 (r34 crop, ReDimNet2); not a large drop.
4. **AHC is brittle, VBx is not.** Centroid-linkage AHC has a cliff at thr >= 0.75 on every model (cpWER 0.43 to 0.55). Leave-one-out exposes it: LOO cpWER is 0.270 (r34 crop) and 0.272 (ReDimNet2) but 0.483 (SimAM-R34) and 0.432 (SimAM-R100) because the 10-speaker file gets a 0.75 pick. ReDimNet2 has the widest plateau (0.6 to 0.725 all within 0.273). Picked thresholds are tuned on 8 files: treat 0.65 to 0.7 as a region, not a number. VBx with crop embeddings needed no tuning.
5. WER is identical in every row (0.214): same words, only attribution differs.

## What was not run / failed
- SimAM `-ft` (Vox2 fine-tuned) variants and IDRnD ReDimNet-B6 (first generation) downloaded but not run: time; the model effect was already flat. Weights fetched fine from HF (`gaunernst/wespeaker-voxblink2-samresnet34`, `lengyue233/wespeaker-voxblink2-samresnet100`, `Wespeaker/wespeaker-voxceleb-redimnet2-B6-LM`).
- A first ReDimNet2 attempt with sub-batch 16 used 6.7 to 7.5 GB and was killed; rerun at sub-batch 4.
- No pretrained embedding was fine-tuned on Mongolian; no PLDA retrained (VBx cannot score SimAM/ReDimNet2 vectors, hence AHC).

## Install notes
- `/mnt/data01/Library/bakeoff-diar/embed/venv`: py3.12, pyannote.audio 4.0.7, torch 2.8.0+cu126, deps pinned from `monsum-speech/.venv` (minus opentelemetry, which conflicts when resolved alone) plus `soundfile`. `uv pip install` needs `--index-strategy unsafe-best-match` with the cu126 extra index.
- `pip install wespeaker` is not on PyPI. Cloned `wenet-e2e/wespeaker` and `IDRnD/ReDimNet` into `embed/`; `wespeaker/__init__.py` imports its CLI (silero_vad) and `frontend/__init__.py` imports s3prl, so the scripts register stub packages in `sys.modules` and import `wespeaker.models.*`, `wespeaker.utils.checkpoint`, `wespeaker.frontend.tfmel` directly. Models load from `avg_model.pt` + `config.yaml`; SimAM uses 80-dim kaldi fbank (x32768, hamming, per-utterance mean norm), ReDimNet2 uses its `tfmel` front-end.
- torchcodec cannot load FFmpeg here; audio is passed in-memory as `{"waveform", "sample_rate"}` like the service does.
- Scripts: `embed/run/extract.py` (stage 1), `cluster.py` (stage 2), `score.sh`, `report.py`, `extract_timing.py`.

## Integrating the winner into `diarization.py`
Nothing to ship from a new model; the lever is the extraction. Candidate with least risk: **keep community-1 weights, VBx, PLDA, threshold 0.6, and change only how `_embedding` is fed** (B3: 0.268, no tuning, speaker count worse on the 40-speaker file, 17 vs 21).
- Code: `get_pipeline()` would wrap `pipeline._embedding` (a `PretrainedSpeakerEmbedding`) in a small object with the same interface (`sample_rate`, `dimension`, `min_num_samples`, `metric`, `__call__(waveforms, masks)`) that crops by mask and calls the model without `weights`. This touches a private pyannote attribute; pin pyannote 4.0.x and add a test. No pipeline-param override can do it (`embedding_exclude_overlap` only changes which mask is used).
- Swapping to SimAM-R100 or ReDimNet2 instead would also need AHC in place of VBx (the checkpoint's `plda/` is tied to ResNet34 embeddings): replace `pipeline.clustering` with `AgglomerativeClustering(metric="cosine")` instantiated with `method=centroid, min_cluster_size=12, threshold=0.65 to 0.7`, plus extra deps (wespeaker model code vendored, since it is not on PyPI) and about 6 GB VRAM. No measurable gain over B3, so not recommended.
- **Voiceprints in monsum-api:** `diarization.py` returns the clustering centroids and tags them with `DIARIZATION_EMBEDDING_MODEL_ID`. B3 keeps the same weights and the same 256-d space, but the vectors are computed differently (crop vs weighted pooling), so they are close but not identical to enrolled ones; bump the model ID and re-enroll rather than assume compatibility (verify cosine between old and new centroids on a few enrolled speakers first). SimAM-R100 is 256-d but a different space; ReDimNet2 is 192-d (pgvector dimension change). Either requires re-enrolling all voiceprints and a new `embedding_model` tag. Under AHC the returned centroids are plain means of cluster members, as with VBx.

## Verdict
- Swapping the embedding model does not help: r34 crop 0.267, SimAM-R100 0.266, ReDimNet2 0.268 are indistinguishable, and none fixes the 40-speaker undercount (25 to 28 of 40).
- The real lever is crop-and-concatenate extraction instead of mask-weighted pooling: about -0.03 cpWER (0.298 to 0.268) with the existing checkpoint, VBx and threshold; worth an ablation (tile-padding vs pooling) and an eval on non-stitched data before changing the service.
- Prefer ReDimNet2 only if swapping models anyway (widest threshold plateau, LOO 0.272); avoid AHC with a single global threshold otherwise.

## Ablation: why B3 (crop + VBx) beats A (0.268 vs 0.298)

Same harness, cached ASR words, community-1 checkpoint/PLDA/VBx untouched, threshold 0.6, no tuning, GPU otherwise idle. Only the embedding step changes. Embedding wall is for the 31.5 min file (embedding only; peak VRAM stays under 2.2 GB for everything except the earlier tile-pad run).
Scripts: `/mnt/data01/Library/bakeoff-diar/embed/run/extract2.py` (modes below), then `cluster.py MODEL vbx 0.6 1`, `score.sh`.

| row | what differs from A | cpWER | spk err | ref>hyp (2/4/40/8/24/10/12/6 spk files) | per-file cpWER | emb wall |
|---|---|---|---|---|---|---|
| A. native | baseline: whole 10 s chunk, fbank centered over the whole chunk, mask-weighted pooling, overlap frames excluded | 0.298 | 3.38 | 2>3, 4>4, 40>21, 8>7, 24>20, 10>10, 12>12, 6>8 | .214 .229 .446 .390 .225 .440 .279 .162 | 43 s |
| B3. crop, tile-pad to whole s | masked samples only, concatenated, tiled to a whole second | 0.268 | 4.12 | 2>2, 4>3, 40>17, 8>6, 24>20, 10>8, 12>11, 6>6 | .201 .223 .396 .366 .184 .393 .223 .159 | 29 to 178 s (contended) |
| 1. crop, no padding | exact masked samples, no padding (length-bucketed; 60 of 34,590 crops were under the model minimum of 400 samples and were tiled up to it) | 0.268 | 4.12 | same as B3 | .201 .223 .395 .364 .185 .393 .223 .158 | 17 s |
| 2. crop, zero-pad to whole s | zeros instead of tiling | 0.273 | 4.12 | 2>3, 4>3, 40>17, 8>6, 24>19, 10>9, 12>12, 6>6 | .210 .223 .387 .364 .199 .411 .221 .157 | 14 s |
| 3. native, fbank centered over masked frames only | A, but the per-chunk fbank mean is the mask-weighted mean (pooling and everything else untouched) | **0.270** | 4.12 | 2>2, 4>3, 40>17, 8>6, 24>20, 10>8, 12>11, 6>6 | .202 .222 .402 .367 .182 .400 .224 .157 | 42 s |
| 3b. native, non-mask audio zeroed | A, chunk audio multiplied by the mask (same length, native pooling) | 0.431 | 4.75 | 2>4, 4>7, 40>19, 8>7, 24>17, 10>11, 12>12, 6>9 | .255 .283 .585 .702 .286 .715 .373 .246 | 43 s |
| 4a. crop, tile-pad, overlap frames NOT excluded | B3 with `embedding_exclude_overlap=False` | 0.270 | 4.12 | same as B3 | .201 .223 .396 .367 .184 .404 .223 .159 | 15 s |
| 4b. native, overlap NOT excluded | A with `embedding_exclude_overlap=False` | 0.301 | 3.62 | 2>3, 4>4, 40>19, 8>7, 24>20, 10>10, 12>12, 6>8 | .215 .229 .477 .388 .226 .438 .279 .162 | 42 s |

Notes on the rows:
- Row "native with tile padding" does not exist: native always embeds a fixed 10 s chunk, nothing is padded. The padding control is 1 vs B3 vs 2: no padding = tile = 0.268; zeros cost 0.005 (they look like silence to the network).
- Mask binarisation: crop uses `mask > 0.5` on the already binary `used_mask`, native uses the same mask nearest-resampled to the ResNet frame rate, so there is no separate binarisation factor. Min active frames is a clustering-side filter (`filter_embeddings`, 20 % of the chunk), identical in every row.
- 3b is not a clean control: zeroed audio gives fbank log(eps) frames that wreck the chunk mean and the convolutions, so it shows the network is very sensitive to what lies outside the mask, nothing more.

**Which factor carries the gain: the fbank mean normalisation (CMN).** `compute_fbank` subtracts the mean over the entire 10 s chunk, so the features of the target speaker are shifted by the average of everyone else, silence and noise in the same chunk. Centering on the target speaker's own frames (row 3) recovers the whole gain (0.298 to 0.270) while keeping native batching, native pooling, no cropping, no padding, and the same speed as A. Cropping (B3, row 1) gets the same number because it implicitly centers on the speaker's own samples. Padding mode (tile, none) and overlap exclusion (about 0.002 to 0.003) are not factors. Speaker count behaves identically in rows 1, 3 and B3 (40>17, mean err 4.12 vs 3.38 for A): the gain is lower WER-attribution error on files already well split, and the 40-speaker file is split into fewer clusters (cpWER on that file still improves, 0.446 to about 0.40).

### How pyannote 4.0.7 builds embeddings (site-packages/pyannote/audio)
- `pipelines/speaker_diarization.py:375-397` builds `clean_segmentations`: frames with 2 or more active speakers are zeroed when `embedding_exclude_overlap` is True (set by the checkpoint config).
- `speaker_diarization.py:417-425`: per (chunk, speaker) it uses the clean mask if it has more than `min_num_frames` active frames, else the raw mask (`used_mask`). `speaker_diarization.py:399-433`: the whole 10 s chunk waveform plus that mask go to `self._embedding(waveform_batch, masks=mask_batch)`.
- `pipelines/speaker_verification.py:704-716` (`PyannoteAudioPretrainedSpeakerEmbedding.__call__`) calls `self.model_(waveforms, weights=masks)`.
- `models/embedding/wespeaker/__init__.py:282-286`: `forward` = `compute_fbank(waveforms)` then `self.resnet(fbank, weights=weights)[1]`. `compute_fbank` (same file, around lines 100-135) does `features - features.mean(dim=1)` over the whole chunk when `fbank_centering_span` is None (the checkpoint default): this is the factor.
- `models/embedding/wespeaker/resnet.py:399-421`: ResNet forward; `weights` only reach `self.pool(out, weights=weights)`. `models/blocks/pooling.py:76-121` (`StatsPool`): weights nearest-interpolated to the ResNet frame count (line 117) and used for weighted mean/std. So the mask affects pooling only, never the convolutions or the centering.

### Recommended port for `monsum-speech/src/monsum_speech/adapters/diarization.py`
Patch the embedding model's `forward` once after loading the pipeline (in `get_pipeline()` before `pipeline.to(device)`). This is the exact code that was run (`native_patch`; embeddings are bit-identical, max abs diff 0.0, to the tested wrapper of row 3, wall 42 s on the 31.5 min file, same VRAM as A):

```python
import types
import torch

def _mask_centered_forward(self, waveforms, weights=None):
    # same as WeSpeakerResNet.forward, except the fbank mean is taken over the speaker's own frames
    feats = torch.vmap(self._fbank)(waveforms * (1 << 15))  # (B, T, 80), uncentered
    if weights is None:  # min_num_samples probing and any unmasked call
        return self.resnet(feats - feats.mean(1, keepdim=True))[1]
    w = weights.float()
    wt = w[:, torch.arange(feats.shape[1], device=w.device) * w.shape[1] // feats.shape[1]]
    mean = (feats * wt[..., None]).sum(1, keepdim=True) / wt.sum(1, keepdim=True)[..., None].clamp(min=1)
    return self.resnet(feats - mean, weights=w)[1]

model = pipeline._embedding.model_
model.forward = types.MethodType(_mask_centered_forward, model)
```
- It relies on `model._fbank` (private) and `fbank_centering_span is None`; pin pyannote.audio 4.0.x and add a unit test that, for an all-ones mask, the patched output equals the stock output (should match to float tolerance) and that for a partial mask it differs.
- Checkpoint, PLDA, VBx and threshold stay as they are; no new dependency, no extra VRAM, no speed cost (unlike the crop wrapper).
- Voiceprints: centroids stay 256-d from the same weights, but the vectors move (different centering), so bump `DIARIZATION_EMBEDDING_MODEL_ID` (e.g. append `:fbank-cmn=speaker`) and re-enroll monsum-api voiceprints; first check old-vs-new cosine on a few enrolled speakers.
- Caveats: gain measured on 8 stitched files only (same ASR words), row 3 improves cpWER on all 8 files (the 4-speaker and 6-speaker files by only 0.005 to 0.007); the 40-speaker file ends with 17 speakers, so the panel undercount is not solved by this.