Finding the best automatic transcription engine for each language

We are comparing several automatic transcription engines side by side, Speechmatics, ElevenLabs, Deepgram, YouTube's own captions, and OpenAI's open-source Whisper running on our own servers, against short human-transcribed clips in each language, to see which one actually gets the words right and times them precisely enough for things like Smart Subtitles to work well.

Transcription quality varies a lot by language: some engines are excellent in widely-spoken languages and much rougher in others, and a wrong or mistimed caption breaks the whole reading-along experience.

As we grew our learner base in some of the smaller languages a lot over the past months, rather than assume which engine is best for each language, we are measuring it properly, language by language, against human-labelled ground truth data so each one gets whichever engine actually performs best for it.

Transcription is a messy field, and there will always be errors that require a human to correct, but we're trying to figure out which provider performs best and gets us 80-90% there, so that less subtitle errors need to be reported and corrected by humans.