FlowtandBriefAI · Software · ResearchMeet Flowtand

Open ASR Leaderboard adds multilingual and long-form tracks

Announcement

Hugging Face

Browse Benchmarks topic page →

A Hugging Face blog post reports that the Open ASR Leaderboard now compares 60+ open and closed-source models across 11 datasets and adds multilingual and long-form transcription tracks. Its cited takeaways: Conformer-plus-LLM decoders lead English accuracy, CTC/TDT decoders are far faster, multilingual coverage trades against single-language accuracy, and closed systems still lead long-form.

What the leaderboard update reports

A Hugging Face blog post published November 21, 2025 says the Open ASR Leaderboard has added two evaluation tracks: multilingual transcription and long-form transcription (meetings, lectures, podcasts). The authors state the board compares 60+ open and closed-source models from 18 organizations across 11 datasets, measuring accuracy via word error rate (WER) and efficiency via inverse real-time factor (RTFx), where higher is better. Their stated takeaways: models pairing a Conformer encoder with an LLM decoder (NVIDIA Canary-Qwen-2.5B, IBM Granite-Speech-3.3-8B, Microsoft Phi-4-Multimodal-Instruct) post the lowest English WER; CTC and TDT decoders are described as 10–100x faster with somewhat higher error rates; Whisper Large v3 remains a strong multilingual baseline across 99 languages; and closed systems still lead long-form. For long-form throughput they cite Parakeet CTC 1.1B at RTFx 2793.75 versus 68.56 for Whisper Large v3, with WER of 6.68 and 6.43 respectively. A commenter noted Parakeet v3 supports 25 languages; the author replied the post should have specified Parakeet v2 and that v3 sits on the main board but not yet the long-form track.

Application scenario: transcription platform engineer

A team building podcast or meeting transcription could use the added long-form track to shortlist candidates by WER and RTFx jointly rather than accuracy alone, since the authors argue throughput becomes a deciding factor at that audio length. For an English-only pipeline with tight latency or batch cost constraints, a CTC-based Conformer such as Parakeet (per the post, English-only in the version benchmarked for long-form) is a candidate worth testing against Whisper Large v3. A product spanning multiple languages could instead treat Whisper Large v3 or a distilled variant as a starting baseline and check the multilingual track for coverage.

Trial steps and how to judge results

A team could run a small bake-off: sample a few hours of its own long audio, transcribe it with two shortlisted models from the leaderboard, and record WER against human-corrected transcripts alongside RTFx on identical hardware. Establish a baseline with the current model first, then compare per-file and aggregate WER and throughput; watch for error accumulation on long inputs and for language gaps. Failure checks: domain shift from leaderboard datasets and mismatched hardware could invert the reported ranking, and a model noted as single-language should be rejected for multilingual inputs. Keep a human review step on outputs before deployment.

Trade-offs, prerequisites and unknowns

The cited pattern is that accuracy-optimized LLM decoders are slower, while fast CTC/TDT models accept moderately higher WER, and that multilingual coverage or English specialization usually costs the other. Prerequisites include representative evaluation audio and the compute to measure RTFx consistently. Unknowns: the excerpt notes only five languages are benchmarked, long-form coverage of newer models is partial, and rankings shift quickly (the Parakeet version question shows this). Do not use the board as a substitute for in-domain testing, and avoid it when the target language or audio type is not represented.

Context and limitations

  • Based on a short source excerpt; the full article was not analyzed.
  • All figures and rankings are as reported by the leaderboard authors, not independently verified.
  • Only five languages are benchmarked in the multilingual track per the post; long-form results for newer models are noted as incomplete.
  • Based on bounded source text, not independent testing. Practical suggestions are editorial proposals.