Multimedia Analysis for Unsupervised Dubbing In Entertainment (MAUDIE)

"With the availability of numerous streaming video services that now operate around the world, feature films and TV series are being enjoyed by much larger audiences than ever before. This has led to a significant surge in demand for high quality, professional services to adapt titles into multiple languages using both subtitling, where translated text is displayed over the picture, and dubbing, where the voices of on-screen actors are replaced by foreign language speakers.

The most popular approach taken by content owners is subtitling, since this is on average 10% of the cost of dubbing the same content. However, consumer audiences are becoming more and more discerning, and the providers of streaming video services are finding that it is increasingly necessary to provide high quality voice dubs for a growing number of countries. Recent research by the Media Entertainment Services Alliance (MESA) indicates that around $1.4bn was spent in EMEA on dubbing in 2016, a figure that is expected to grow at a rate of more than 10% annually.

Traditionally, dubbing has been fulfilled predominantly by independent, dedicated dubbing studios that operate in each country. These provide recording spaces, control rooms, equipment and a staff of in-house specialists to work on projects. Each such studio engages the services of voice actors, most of whom reside nearby and can therefore attend recording sessions without incurring significant travel and subsistence expenses.

With the recent growth in demand, the entertainment industry now faces a significant shortfall in capacity and talent used to create foreign language dubs due to the limited availability of these dedicated dubbing studios and their key staff.

In 2017, ZOO Digital, a partner in the consortium and established provider of subtitling services, began offering a new disruptive approach to dubbing that utilises cloud computing to significantly increase industry capacity. Despite this, there remains a fundamental constraining factor due to the limited availability of experienced _dubbing directors_ who are needed to oversee each recording session.

This project will use recent advances in machine learning research in order to automate much of the role of the dubbing director. This will mean that voice actors will be able to perform their work unsupervised, with critical appraisal of their timing accuracy and performance provided in near-real time by automation software. This will pave the way for a substantial increase in capacity for high quality dubbing to meet the expanding industry requirements."