How often have we seen movies, TV shows, and music videos in an attempt to be more PG-friendly have certain words censored, muted, or dubbed over? While it probably won’t affect the story much in general, it does make it rather awkward especially since the replaced words don’t really line up with the character’s lips on screen.
This is something that Disney Research has noticed which is why they are now looking at ways to make it more realistic. What they have come up with so far is an automated system which basically generates alternative dialog while keeping the words in sync with the person’s lips. This relies on the words or sentences having similar phonemes, which is basically the smallest form of speech that differentiates two words, like “pad” and “pat”, “bad” and “bat”.
This is then combined with dynamic visemes (which are basically visual phonemes) to create a somewhat believable dub. According to Disney Research, “When composed with a pronunciation dictionary and language model, this produces a vast number of word sequences that are in sync with the original video, literally putting plausible words into the mouth of the speaker.”
In the video above they demonstrate how this works and while it does look plausible, we guess the robotic voice kind of ruins it, but perhaps this is just the first step towards creating more believable dubbed TV shows, movies, and music videos.
Filed in . Read more about Disney.