ElevenLabs is building a universal AI dubbing machine
The system will replace any actor's voice with their same voice, just in a different language.
After Disney releases a new film in English, the company will go back and localize it in as many as 46 global languages to make the movie accesible to as wide an audience as possible. This is a massive undertaking, one for which Disney has an entire division — Disney Character Voices International Inc — to handle the task. And it's not like you're getting Chris Pratt back in the recording booth to dub his GotG III lines in Icelandic and Swahili — each version sounds a little different given the local voice actors. But with a new "AI dubbing" system from ElevenLabs, we could soon get a close recreation of Pratt's voice, regardless of the language spoken on-screen.
ElevenLabs is an AI startup that offers a voice cloning service, allowing subscribers to generate nearly identical vocalizations with AI based on a few minutes worth of audio sample uploads. Not wholly unsurprising, as soon as the feature was released in beta, it was immediately exploited to impersonate celebrities, sometimes even without their prior knowledge and consent.
The new AI dubbing feature does essentially the same thing — in more than 20 different languages including Hindi, Portuguese, Spanish, Japanese, Ukrainian, Polish and Arabic — but legitimately, and with permission. This tool is designed for use by media companies, educators and internet influencers who don't have Disney Money™ to fund their global adaptation efforts.
ElevenLabs asserts that the system will be able to not only translate "spoken content to another language in minutes" but also generate new spoken dialog in the target language using the actor's own voice. Or, at least, a AI generated recreation. The system is even reportedly capable of maintaining the "emotion and intonation" of the existing dialog and transferring that over to the generated translation.
"It will help audiences enjoy any content they want, regardless of the language they speak," ElevenLabs CEO Mati Staniszewski said in a press statement. "And it will mean content creators can easily and authentically access a far bigger audience across the world."
