Open ASR Leaderboard Adds Its First Indian-Language Speech Recognition Test

Voice Arena and Hugging Face have added two new evaluation sets, called Monsoon en-IN and Monsoon hi-IN, to the Open ASR Leaderboard, which ranks automatic speech recognition (ASR) systems. Hindi, spoken by more than half a billion people, is the first Indic language added to a multilingual leaderboard tab that until now covered only European languages.

The leaderboard's usual measure, word error rate (WER), reduces a system's performance to a single number. Past research has found that ASR error rates are not spread evenly: one study found commercial systems were roughly twice as inaccurate for Black speakers as for white speakers, and another found further gaps by gender, age and accent. Those disparities do not show up on a leaderboard because standard test sets record what was said but almost nothing about who said it.

The new Monsoon sets were built to vary along nine factors — including geography, age, gender, vocabulary, devices and acoustic environment — so that errors tied to a particular group of speakers can be detected rather than averaged away. Together, the four public and private splits (two each for Hindi and Indian English) comprise 4,888 speakers, with 12 attributes recorded for each speaker.

The Indian English data was collected from 428 native districts across 24 states and union territories, and the Hindi data from up to 295 districts. In both languages, more than half of all speakers appear in only a single audio clip, and the ten largest contributors account for no more than 6.8% of total recorded duration in any split — meaning no individual voice can dominate a system's score.