The Worst Dubbing to Date Was Done by an AI for Prime Video. Humans Are Upset

We’ve been hearing about how artificial intelligence will improve audiovisual creation to levels never seen before for weeks now. A recent example is its excellent use to make actresses Anya Taylor-Joy and Alyla Browne, who play the protagonist of Furiosa at different ages, look alike. But not all cases are equally fortunate.

AI in dubbing. Although dubbing is an essential process in for movies and shows in non-English-speaking countries, the globalization of content has changed with the rise of streaming. Viewers want to watch movies and series not just in English,  but in their own language. That’s why Korean and Spanish are now commonplace on platforms all over the world. Take Squid Game, for example, which remains the most-watched series in Netflix’s history. It was filmed in Korean, but is available in a variety of languages.

Why use AI? The problem is that not all languages are as in demand as Korean or Spanish, and some require a lot of money for translation, dubbing, quality control, and other details. AI can save money and help studios dub content in even more languages. As such, it has led to the creation companies like Deepdub, Papercup, Resemble AI, and ElevenLabs, which offer AI-powered dubbing services.

Lip movement. In addition to dubbing, studios are looking into using generative AI to adjust the lip movements of actors and actresses to match different languages, as well as to adapt scripts to suit audiences of different ages. This topic is related to other issues, such as deep fakes, which productions use to make actors look younger. According to Variety, specialized agencies like LipDub are adding clients on less demanding platforms such as YouTube, where they already have a hundred customers. Still, LipDub is getting ready to leap Hollywood.

Catastrophic results. You can't get to the big stage before going trial and error, though. One disaster involved green lighting products that are, at the very best, half-baked. Recently, social media users have shared clips My Man is Cupid, True to Love, and Heartbeat, a series of K-dramas on Prime Video’s. The shows were not only released in Korean with subtitles but also dubbed into Spanish—with a neutral accent reminiscent of other eras of Latin American dubbing—using AI. As some users on X have pointed out, the results are catastrophic: the dubs feature lackluster intonations and an absolute mismatch between image and sound, among other failures.

Dubbers are upset. Human dubbing actors and unions have made proposals in response to the situation. A letter signed by Scarlett Johansson triggered some of these demands, as she opposed OpenAI using her voice for Sky, its AI assistant. Actor Masumi Mutsuda says the dubbing unions demand traceability and transparency: They want to know where the voices used by the AI models came from and want companies to stop using voices from productions to train their voice models.

Image | Prime Video

Related | This Movie Is Blowing Up on Netflix, But It's Not From Any of the Usual Countries