Nvidia's new 100M-parameter model tracks eight speakers and tops a diarisation ranking

Nvidia has published Nemotron 3 Diarization, an open-weight model of 100 million parameters that works out who is speaking when, and it handles up to eight speakers where the Nvidia checkpoint it is measured against stopped at four. In VoiceArena's first Diarization-Bench results it ranked first among 12 systems with a diarisation error rate of 14.72%, against 19.3% for the next-ranked system, according to Nvidia's post published alongside the model.

Nvidia's chart of full-set error rate at each input buffer latency setting, with the older four-speaker baseline for comparison. Image: Nvidia

Nvidia's model card gives a release date of September 23 and licenses the weights under version 1.1 of the OpenMDW License Agreement, for commercial or non-commercial use. Diarisation labels time, not identity: the eight output channels are ordered by when each voice first arrives and the labels are generic, so the model can report that speaker1 talked between two timestamps without establishing who speaker1 is.

The ranking is Nvidia's own, and provisional

The VoiceArena result covers 12 systems and 17 system configurations across 139 English-language conversations totalling about 22 hours, scored with overlapping speech included, system-generated speech activity detection and no boundary collar. Nvidia says the model also ranked first with 100 ms and 250 ms collars, and across both in-person and online recordings.

Nvidia attaches a caveat to all of it: "These initial results may change as Voice Arena completes its Version 1 evaluation and paired statistical analysis."

Error rates fall against Nvidia's own four-speaker model

At the 30.4-second offline setting, Nvidia reports a diarisation error rate of 12.73% on the full DIHARD III evaluation, 9.10% on CALLHOME-Part2, 6.40% on AliMeeting Test Near and 6.77% on NOTSOFAR1 MHM. The previous four-speaker streaming Sortformer checkpoint, run as the baseline at the same setting, scores 19.09%, 10.32%, 11.57% and 21.77% on those four sets in that order.

Nvidia's own summary of the gap is a 41.0% mean relative error reduction at the 1.04-second latency setting, which it describes as an unweighted average of the eight per-dataset reductions rather than a pooled score computed across every recording. Those reductions run from 9.0% on CALLHOME-Part2 to 65.2% on NOTSOFAR1 MHM.

NOTSOFAR1 MHM is a meetings benchmark with three to seven speakers on a mix of headset microphones. At the 30.4-second setting the older model got the number of speakers in a recording right 35.00% of the time, against 93.75% for the new one.

One figure runs the other way, and Nvidia prints it. On the two-speaker subset of CALLHOME-Part2, at that same 30.4-second setting, the new model records 5.98% against the baseline's 5.68%, even though the full CALLHOME-Part2 result improves from 10.32% to 9.10%.

Latency is a dial, and 80 milliseconds is not the recommended end

A single checkpoint covers input buffer latencies of 30.4 seconds, 1.04 seconds, 0.64 seconds and 0.32 seconds. Nvidia says it supports a buffer "as low as 80 ms for latency-critical applications, although the lowest recommended configuration is 0.32 s", and that the figure counts only the audio buffered before the model runs, not computation time.

Accuracy gives up less than the latency figures suggest. DIHARD III full error rises from 12.73% at the 30.4-second setting to 13.55% at 0.32 seconds, which is still below the baseline's 19.85% at the same 0.32 seconds.

Underneath is a 31-layer Transformer encoder with rotary positional embeddings, running on 80-millisecond frames, with an Arrival-Order Speaker Cache and a FIFO queue to carry speaker identities across chunks. It runs through Nvidia's NeMo framework, natively in Hugging Face Transformers, and through a C++ runtime, NeMo-Speech.cpp, for diarisation on a local machine.

Nvidia ties the model to no device

Nothing Nvidia published connects this to any product. Live captions on smart glasses transcribe words without saying who owns them, and a group conversation is exactly where that falls apart. Meta has been selling audio capability hard, from Dolby Atmos video capture on its coming AI glasses to hearing enhancement sold through Meta One, and none of that assigns a sentence to a person.

Nvidia trained the model on roughly 10,000 hours of real conversations plus 82,611 hours of simulated multi-talker mixtures, on eight nodes of eight A100-SXM4-80GB GPUs.

One more warning belongs with the numbers. Nvidia scored these results against reference annotations it published alongside the model, and says that changing those labels changes the measured result.