Yandex added simultaneous translation of live broadcasts to the browser

“Yandexadded automatic voice-over technology for YouTube live broadcasts to its browser, the company said.

It is clarified that while the technology is working in the open beta testing mode, therefore it does not work for all broadcasts. In particular, the translation is available for Apple, NASA and Elon Musk’s SpaceX channels, as well as a platform for TED speakers and others. Using this technology, it will be possible to watch a presentation of new technical devices or a broadcast of a rocket launch without waiting for the video to be released in Russian.

The company noted that “when working with a streaming scenario, the minimum delay is important, which means that you need to translate immediately,” but at the same time it is necessary to preserve the context. As a result, Yandex engineers were able to teach neural networks to translate the speech of speakers without waiting for them to finish the sentence, without losing the meaning.

According to the head of the Yandex and Yandex.Browser application, Dmitry Timko, in order to achieve this, it was necessary to completely rebuild the architecture of video off-screen translation. In particular, one neural network recognizes audio and turns it into text, another one determines the gender of the speaker by biometrics, the third one places punctuation marks and highlights semantic fragments, and the fourth one translates.

Now neural networks are able to translate broadcasts in five languages ​​- English, German, French and Italian. It is assumed that in the future the technology will learn to translate broadcasts in Chinese and Japanese, as well as new pairs of European languages ​​will be added. In addition, the developers of the algorithm hope to add translation of streaming videos and broadcasts to other platforms, including Twitch, a resource popular with gamers.

Subscribe on Vedomosti on Telegram and stay up to date with the main economic and business news.