FlowtandBriefAI · Software · ResearchMeet Flowtand

Open ASR Leaderboard Adds Egyptian and Nigerian English Private Splits

Announcement

Hugging Face

Browse Benchmarks topic page →

Hugging Face's Open ASR Leaderboard added two private conversational English splits from Liva AI, covering Egyptian and Nigerian accents. The additions expand accent coverage and let the leaderboard report a breakdown based on Kachru's Three Circles, comparing Inner Circle accents against Outer and Expanding Circle ones.

What the leaderboard added

Hugging Face's Open ASR Leaderboard, which already carried private splits for several accents, announced new private data from a collaboration with Liva AI covering Egyptian and Nigerian English. The Egyptian split runs 1.34 hours and the Nigerian split 1.55 hours; both are described as conversational and spontaneous, with transcripts that are punctuated, cased and include disfluencies. The stated goals are better accent representation in ASR evaluation and more trust through private data that resists overfitting to public test sets. Alongside the data, the leaderboard now reports a richer accent breakdown borrowed from Kachru's Three Circles: US versus non-US accents, and Inner Circle (US, UK, Australia, Canada) versus Outer and Expanding Circles (India, Nigeria, Egypt). The excerpt does not report model scores on the new splits.

A scenario for an evaluation lead

An evaluation lead at a company deploying voice transcription in customer-support or call-center settings could use these splits to check whether a shortlisted ASR model holds up on Egyptian and Nigerian English, rather than relying only on aggregate English word error rates. Because the transcripts preserve casing, punctuation and disfluencies, a team could compare raw transcription and a post-processed pipeline against the same reference, keeping the comparison consistent with how the leaderboard frames its private splits. The Kachru breakdown is also a useful framing device: it gives stakeholders a vocabulary for distinguishing Inner Circle accent performance from Outer and Expanding Circle performance, which may matter if the deployment population includes speakers from those regions.

How to trial it

A team could first record its current baseline on its own audio for Egyptian and Nigerian English speakers, using the same normalization rules as its production pipeline. Then it could compare candidate models on the leaderboard's new private splits, treating those scores as one signal rather than a verdict. Success could be defined as a smaller gap between Inner Circle and Outer/Expanding Circle performance than the baseline shows, measured on identical preprocessing and reference conventions. Failure checks should include inspecting errors on disfluent or spontaneous speech, since both new splits are conversational; if a model degrades mainly there, a team should treat that as a targeted finding rather than a general ranking. Human review of a sample of transcripts remains advisable before any deployment decision, and the excerpt does not establish that these splits predict performance on other domains.

Trade-offs and unknowns

Private splits reduce benchmark gaming but limit reproducibility: outsiders cannot inspect the audio, so they depend on the maintainers' reporting and cannot independently verify conditions. The new splits are small (about 1.3 to 1.6 hours each), so per-accent scores may be noisy, and the excerpt does not describe speaker counts, recording conditions or domain. Teams should not assume these splits cover all Egyptian or Nigerian English speech, nor that leaderboard improvements transfer to their own telephony or noisy environments. They are best used as a conditional cross-check alongside in-house evaluation, not as a substitute for it. The excerpt also gives no dates for score updates, so readers should check the leaderboard itself for current results.

Context and limitations

  • Based on a short source excerpt; the full article was not analyzed.
  • The source gives split durations and transcription conventions but no per-model results for the new accents.
  • Private data is intended to reduce benchmark overfitting, but the excerpt does not describe its collection methodology in detail.
  • Based on bounded source text, not independent testing. Practical suggestions are editorial proposals.