Pandemonium

Who said what, in a whole meeting, in one pass. Pandemonium is a long-form, diarization-conditioned multi-talker ASR model from BUT Speech@FIT. Given a recording and a diarization of it, it returns a time-stamped transcript for every speaker, decoding all speakers of the conversation together so that each speaker's transcript is informed by the others.

It is described in Pandemonium: Longform Target-Speaker Speech Recognition with Cross-Speaker Interactions (Polok et al., submitted to ICASSP 2027; PDF). Please refer to the paper for the method, the training recipe and the full evaluation, and cite it if you use the model. Code: BUTSpeechFIT/Pandemonium.

Pandemonium: the collective noun for a flock of parakeets — or chaos.

Task	speaker-attributed (multi-talker) ASR, English
Input	mono 16 kHz audio + per-speaker activity from any diarizer (soft probabilities or hard segments)
Output	one time-stamped word sequence per speaker
Backbone	Parakeet-TDT-0.6b-v3 FastConformer encoder (24 layers, d=1024), local attention [256, 256] + 4 global tokens
Head	frozen Parakeet joiner used as a frame-synchronous CTC head, greedy decoding
Parameters	≈0.7 B
Speakers	up to 8 per recording
Context	whole recordings in one pass (evaluated on meetings of up to 1 h)
Speed	RTFx 426× including Nemotron-3-Diarization, 690× ASR alone (released pipeline, below)

How it works

Pandemonium builds on diarization-conditioned target-speaker ASR (DiCoW, SE-DiCoW). For each speaker found by a diarizer, the model is conditioned on a frame-level STNO mask — the probabilities that a frame is silence, target only, non-target only, or overlap — which modulates every encoder layer through class-specific affine transforms (FDDT). One recognizer is thereby turned into a recognizer of that speaker, while remaining aware of the others.

On top of that, Pandemonium adds three things:

1. Long-form backbone (DiCoP). The conditioning is moved from Whisper's fixed 30 s windows into Parakeet's FastConformer, whose local attention processes entire meetings on a single GPU. Because the TDT transducer loss does not scale to multi-minute targets, the model is trained with CTC through the (frozen) TDT joiner with its predictor contribution zeroed — no new parameters.
2. Cross-speaker attention (CSA). A conversation's speakers are encoded as one batch, and after encoder layers 8, 12, 16 and 20 each frame attends across the speaker axis (multi-head attention, no positional encoding, so the number of speakers is not fixed). Speakers are no longer decoded in isolation: overlaps, turn-taking and backchannels can be resolved jointly.
3. STNO mask refiner. After layers 12, 16 and 20 the model re-estimates each speaker's activity from the cross-speaker-informed states, rebuilds the STNO mask from it, and blends it into the conditioning of the deeper layers through a learned gate. This lets it repair diarization errors, and separate speakers that the input mask labels identically in overlapped frames.

Both new blocks start as near-identities, so training warm-starts cleanly from Parakeet.

Training

Three stages, each starting from the previous one:

stage	data	hours
1. synthetic	FastMSS mixtures of up to 8 LibriSpeech speakers (2k h) + Fisher conversations restored with Sidon and force-aligned (2k h)	4.0k
2. pseudo-labeled	multi-speaker YODAS2 conversations, diarized with streaming Sortformer and transcribed by DiCoW v3.3, filtered for language ID, hallucinated repetitions and cross-speaker leakage	4.6k
3. real	NOTSOFAR-1 and AMI single-distant-microphone training sets	115

Training conversations are capped at 8 speakers. The diarization mask is randomly corrupted during training so that the refiner has errors to correct; the refiner is trained at a 10× learning rate. This checkpoint is the best epoch by validation cpWER (epoch 16, cpWER 0.1597). The exact training configuration is included as pandemonium.yaml (recipe) and model_config.yaml (the configuration stored in the checkpoint).

Training data and licences

source	used for	licence
Parakeet-TDT-0.6b-v3	initialization (encoder, joiner, tokenizer)	CC-BY-4.0
LibriSpeech	stage 1, synthetic mixtures	CC-BY-4.0
Fisher English (LDC2004S13, LDC2005S13)	stage 1, restored with Sidon (MIT, preprocessing only)	LDC user agreement
YODAS2	stage 2, pseudo-labeled	CC-BY-3.0
NOTSOFAR-1	stage 3	CC-BY-4.0
AMI	stage 3	CC-BY-4.0

No training audio or transcripts are redistributed with the model.

Results

All numbers are cpWER or tcpWER in %, lower is better, with greedy decoding, full recordings, and whisper_nsf text normalization. NSF = NOTSOFAR-1, Mx6 = Mixer 6, SBC = Santa Barbara Corpus (conversation and task subsets with 2–8 speakers); M1 and M2 are two further meeting corpora, not named here for licensing reasons. NOTSOFAR-1 and AMI are in-domain; the rest are out-of-domain. Commercial APIs marked "API A–D" are anonymized.

Paper (cpWER)

system	NSF	AMI	M1	Mx6	SBC	M2	Avg
Oracle diarization
DiCoW v3.3	15.5	14.5	14.1	11.4	15.0	27.1	16.3
SE-DiCoW	15.3	14.4	14.3	11.7	15.2	26.2	16.2
DiCoP (no CSA)	17.3	14.5	14.4	11.9	16.0	26.1	16.7
Pandemonium	14.1	13.4	13.3	11.1	13.8	22.4	14.7
+ MOVER (all distant mics)	10.8	12.6	11.9	9.7	13.8	19.4	13.0
Predicted diarization (pyannoteAI Precision-3)
DiCoW v3.3	23.0	17.3	18.6	11.5	19.3	30.0	20.0
SE-DiCoW	22.8	17.3	18.2	11.8	19.6	29.3	19.8
DiCoP (no CSA)	23.2	16.9	18.0	12.1	19.4	28.9	19.8
Pandemonium	22.4	16.8	18.1	11.7	18.6	27.4	19.2
+ MOVER (all distant mics)	19.0	14.5	15.7	10.2	18.6	24.8	17.1
best published single-mic (Sep 2026)	22.8	17.6	20.9	14.5	–	34.0	–

With predicted diarization, Pandemonium is below the best published single-microphone result on every set where one exists.

With Nemotron-3-Diarization (the released pipeline)

Conditioning on the soft speaker probabilities of Nemotron-3-Diarization (offline setting, no thresholding), on 251 full recordings, 54.5 h:

	NSF	AMI	M1	Mx6	SBC	M2	Avg
cpWER	17.93	16.14	14.53	12.03	18.67	23.52	17.14
tcpWER (5 s collar)	18.23	16.76	14.82	12.28	19.09	24.90	17.68

Soft conditioning beats thresholded segments from the same diarizer (NOTSOFAR-1: 17.93 vs 18.52 cpWER), even though the model was trained on hard masks only.

Comparison with other systems (October 2026)

Same 251 recordings, references, normalization and scorer as the table above; tcpWER, 5 s collar. Every system uses its own diarization unless stated.

system	NSF	AMI	M1	Mx6	SBC	M2	Avg	RTFx
Pandemonium + Nemotron-3-Diarization	18.23	16.76	14.82	12.28	19.09	24.90	17.68	426×
Pandemonium + pyannoteAI Precision-3	23.41	17.85	18.90	11.93	19.14	29.89	20.19	95× API ∥
SE-DiCoW + DiariZen	25.83	19.28	20.73	12.20	21.58	35.98	22.60	28.9×
Nemotron-3-Diarization + MT-Parakeet §	31.56	20.38	12.93 §	18.56	26.83	35.65	24.32	~60×
MOSS-Transcribe-Diarize	28.10	32.14	25.60	11.15	22.48	39.90	26.56	6.1×
API A	34.11	28.65	31.31	12.36	24.31	29.67	26.74	1.0× API ¶
API B	34.84	27.41	32.73	14.24	24.05	31.61	27.48	67× API
Xiaomi CocktailASR + DiariZen	32.78	23.19	28.33	16.18	26.90	38.45	27.64	8.2×
MAI-Transcribe-2 ‡	32.97	38.06	39.32	10.91	28.44	28.59	29.72	214× API
ElevenLabs Scribe v2	37.41	35.54	43.76	9.63	24.50	36.43	31.21	81× API
API C	42.53	34.40	34.56	16.16	32.10	42.16	33.65	141× API
VibeVoice-ASR	36.33	41.88	42.48	12.01	29.53	64.73	37.83	6.6×
Gemini 3.5 Transcribe †	47.75	45.33	52.03	21.27	41.37	48.46	42.70	49× API
API D	52.86	39.37	31.19	25.42	37.88	77.24	43.99	224× API
Mistral Voxtral ‡	58.04	42.95	46.04	19.56	39.03	63.62	44.87	112× API
OpenAI gpt-4o-transcribe-diarize	56.42	60.07	59.99	28.11	45.76	62.66	52.17	2.7× API

• † Gemini is scored with cpWER, as its word timestamps drift on long audio. It is run on windows of up to 15 min with its own speaker labels, linked across windows through DiariZen.
• ‡ Returns only segment-level timestamps, which costs it under tcpWER.
• § Its training data may include the M1 evaluation sessions.
• ¶ Streaming API, which accepts audio no faster than real time.
• ∥ Word error rates use the Precision-3 diarization of the paper, conditioned as hard segments, decoded as full recordings (cpWER 19.14 average, matching the paper's 19.2 above). RTFx is the public pyannoteAI API (92.7 s for upload and diarization, client wall-clock) plus Pandemonium on one GPU (12.6 s); the public API's output differs slightly from the paper's diarization (11.5% DER between the two).
• RTFx is audio duration divided by processing time, including diarization, on one median-length recording per set (6 recordings, 10,022 s). Local models are timed on one GPU with warm-up excluded; API figures are client wall-clock times including upload, and are not directly comparable.
• SBC uses 44 of its 60 sessions.

Headset mixes

The individual headset (IHM) channels of a meeting summed into one signal: overlapped speech without far-field reverberation. AMI uses the official Mix-Headset recordings; NOTSOFAR-1 has none, so its close-talk channels are summed. Same references and scorer as above; tcpWER / cpWER, 5 s collar.

system	NSF-IHM-mix	AMI-IHM-mix	Avg
Pandemonium + oracle activity	8.84 / 8.81	9.71 / 9.59	9.28 / 9.20
Pandemonium + Nemotron-3-Diarization	12.33 / 12.14	11.46 / 11.16	11.90 / 11.65
SE-DiCoW + DiariZen	17.68 / 16.90	15.12 / 14.42	16.40 / 15.66
DiCoW v3.3 + DiariZen	17.83 / 17.05	15.20 / 14.48	16.52 / 15.77
Nemotron-3-Diarization + MT-Parakeet	20.20 / 19.71	13.83 / 13.37	17.02 / 16.54
Xiaomi CocktailASR + DiariZen	23.09 / 22.07	15.89 / 15.17	19.49 / 18.62
MOSS-Transcribe-Diarize	23.90 / 23.63	20.08 / 19.72	21.99 / 21.68
VibeVoice-ASR	26.38 / 25.94	23.71 / 23.16	25.05 / 24.55

Short fully-overlapped mixtures

Read speech mixed at full overlap, 10–20 s per mixture: LibriSpeechMix (test-clean, 1–3 speakers) and LibriMix (test-clean, 2–3 speakers, clean and with WHAM! noise). cpWER; tcpWER is nearly identical at these lengths.

system	LSMix 1	LSMix 2	LSMix 3	L2Mix	L2Mix noisy	L3Mix	L3Mix noisy	Avg
SE-DiCoW + DiariZen	1.82	3.13	21.94	3.49	9.88	26.61	35.09	14.57
Pandemonium + Nemotron-3-Diarization	2.34	3.20	5.17	6.76	17.81	28.02	41.76	15.01
DiCoW v3.3 + DiariZen	1.89	3.65	22.63	4.04	10.50	28.45	36.27	15.35
Pandemonium + oracle activity	2.27	3.37	5.48	10.19	23.25	33.73	46.62	17.84
Xiaomi CocktailASR + DiariZen	1.93	5.41	19.57	13.53	29.36	36.16	50.93	22.41
MOSS-Transcribe-Diarize	3.59	11.29	19.06	24.82	36.19	55.89	64.52	30.77
Nemotron-3-Diarization + MT-Parakeet	2.63	7.43	13.01	24.68	43.79	62.19	73.31	32.43
VibeVoice-ASR	2.18	50.90	73.79	57.87	64.81	82.06	87.83	59.92

Oracle activity is rasterized from the reference segments, which on LibriMix are looser than a diarizer's output; this is the likely reason it trails the Nemotron-conditioned run there.

Single-speaker speech

Multi-talker training costs accuracy on ordinary single-speaker speech. Open ASR Leaderboard test sets (TED-LIUM is no longer distributed with them), WER in %, the leaderboard's Whisper English normalizer. Parakeet-TDT-0.6b-v3 is the base model, run by us (matching its leaderboard scores).

system	LS clean	LS other	VoxPopuli	Earnings-22	AMI-IHM	GigaSpeech	SPGISpeech	Avg
Parakeet-TDT-0.6b-v3	1.94	3.59	6.10	11.18	11.34	9.53	3.98	6.81
Pandemonium, whole recording as one speaker	2.30	5.07	15.57	16.00	13.76	10.26	5.59	9.79
Pandemonium + Nemotron-3-Diarization	2.35	5.00	15.26	16.07	14.13	10.89	5.54	9.89

Most of the gap is on VoxPopuli: greedy CTC decoding through the frozen TDT joiner drops word-internal subword pieces ("suffering" → "suering", "crisis" → "cr"). Pandemonium also outputs lowercase text without punctuation. For single-speaker audio, Parakeet-TDT-0.6b-v3 remains the better choice.

Speed

• Paper: RTFx 330 for the ASR alone, against 405 for the same backbone without the interaction modules and 4–8× slower DiCoW/SE-DiCoW (Fig. 2 of the paper). Measured with the research decoding code, for comparing architectures; the released inference code is faster.
• Released pipeline: RTFx 426 including Nemotron-3-Diarization; on the 6 timing recordings above, 9.0 s of diarization and 14.5 s of transcription for 167 min of audio (690× for the transcription alone). The whole NOTSOFAR-1 eval set (16.7 h) takes 117 s to diarize and about 3 min to transcribe.
• With pyannoteAI Precision-3 (API): RTFx 95 on the same recordings, 92.7 s in the API and 12.6 s of transcription.
• Memory: a 60-minute, 8-speaker recording peaks at about 12 GiB of GPU memory in bf16.

Usage

The model needs a diarization of the recording. The reference pipeline uses Nemotron-3-Diarization and conditions on its soft speaker probabilities. The inference code is a small self-contained package (pandemonium/ + pipeline.py, in demo/ of the code repository).

conda create -n pandemonium python=3.12 -y && conda activate pandemonium
pip install Cython packaging
pip install -r requirements.txt   # pins NeMo to a main-branch commit; Nemotron-3-Diarization needs it
hf auth login                     # access to this gated repo

import librosa
from pipeline import DemoPipeline, utterances

pipe = DemoPipeline()                       # downloads Pandemonium + Nemotron-3-Diarization
audio, _ = librosa.load("meeting.wav", sr=16000, mono=True)

diar = pipe.diarize(audio)                  # per-speaker probabilities, up to 8 speakers
words = pipe.transcribe(audio, diar)        # soft conditioning; soft=False uses thresholded segments
for u in utterances(words):                 # words grouped at pauses > 1 s, interleaved by time
    print(f"[{u['start']:7.1f}-{u['end']:7.1f}] speaker {u['speaker']}: {u['text']}")

To use a different diarizer, pass its per-speaker activity at the model's 80 ms frame rate directly:

from pandemonium import Pandemonium

model = Pandemonium.from_pretrained()        # bf16 on CUDA by default
T = Pandemonium.num_frames(len(audio))
activity = ...                               # (num_speakers, T) array in [0, 1]: probabilities or 0/1
words = model.transcribe(audio, activity)    # list (per speaker) of Word(text, start, end)

Files

file	contents
cp_wer=0.1597-epoch=16.ckpt	weights (PyTorch Lightning checkpoint: state_dict + hyper_parameters, fp32, 2.8 GB)
model_config.yaml	the configuration stored in the checkpoint
pandemonium.yaml	training recipe
tokenizer.model	SentencePiece tokenizer, shared with Parakeet-TDT-0.6b-v3
Pandemonium_ICASSP2027.pdf	the paper

When loading the checkpoint in the training code rather than the inference package, set model.num_speaker_slots=8 +model.encoder.subsampling_conv_chunking_factor=2.

Limitations

• English only. The backbone is multilingual, but the real and evaluation data are English; other languages are untested. Mandarin is not supported by the Parakeet tokenizer.
• Depends on the diarization. Missed speakers are not transcribed, and merged speakers are transcribed as one. Pandemonium's refiner corrects mask errors but cannot add speakers.
• At most 8 speakers per recording.
• Single channel. Multi-microphone recordings are best decoded per microphone and combined with MOVER (see the paper).
• Trained on hard masks. Soft diarizer probabilities work better in practice (see above), but they are outside the training distribution.
• Greedy decoding only; verbatim style (fillers and backchannels are transcribed), which may differ from references that drop them.

Citation

If you use Pandemonium, please cite the paper:

@inproceedings{polok2027pandemonium,
  title     = {Pandemonium: Longform Target-Speaker Speech Recognition with Cross-Speaker Interactions},
  author    = {Polok, Alexander and Klement, Dominik and Medennikov, Ivan and Laurent, Antoine and
               P{\'a}lka, Petr and Cornell, Samuele and Watanabe, Shinji and Khudanpur, Sanjeev and
               Balam, Jagadeesh and {\v{C}}ernock{\'y}, Jan and Burget, Luk{\'a}{\v{s}} and Wiesner, Matthew},
  booktitle = {Submitted to ICASSP},
  year      = {2027}
}

Acknowledgements

This work was supported by JSALT 2026 at Johns Hopkins University (NSF CCRI Grant No. 2120435, Google DeepMind, JHU HLTCOE, JHU AI2AI, ACL), by the Ministry of Education, Youth and Sports of the Czech Republic through the OP JAK project "Linguistics, Artificial Intelligence and Language and Speech Technologies: from Research to Applications" (CZ.02.01.01/00/23_020/0008518), and by the Brno Ph.D. Talent Scholarship Programme. Computational resources were provided by e-INFRA CZ (ID: 90254) and by IDRIS under GENCI allocations AD011012527, A0201012177 and AD011016176.

License

CC-BY-4.0. Pandemonium is a fine-tuned derivative of Parakeet-TDT-0.6b-v3 (CC-BY-4.0, NVIDIA); see Training data and licences for the data it was trained on.

Contact

ipoloka@fit.vut.cz · BUT Speech@FIT, Brno University of Technology