MaDrIgAL: MultiDimensional Interaction management and Adaptive Learning

As tech giants like Google, Facebook, Apple and Microsoft continue to invest in speech technology, the global voice recognition market is projected to reach a value of $133 billion by 2017 (companiesandmarkets.com, 2015). Speech-enabled interactive systems in particular, such as Apple's Siri and Microsoft's Cortana, are starting to show significant economic impact, with the virtual personal assistant (VPA) market estimated to grow from $352 million in 2012 to over $3 billion in 2020 (Grand View Research, 2014).

Although such commercial systems allow consumers to use their voice in interacting with their devices and services, the user experience is still limited due to the lack of naturalness of the conversations and limited social intelligence of the VPA. Moreover, the quality of these user interfaces relies on large, carefully crafted rule sets, making development labour-intensive and not scalable to new application domains. With the emergence of the Internet of Things and voice control in the smart home, there is a huge demand for scalable development of natural conversational interfaces across task domains.

MaDrIgAL will develop a radically new approach to building interactive spoken language interfaces by exploiting the multi-dimensional nature of natural language conversation: in addition to carrying out the underlying task or activity, participants in a dialogue simultaneously address several other aspects of communication, such as giving and eliciting feedback and adhering to social conventions. In analogy to the singing voices in a madrigal, simultaneous processes for each dimension operate in harmony to produce multifunctional, natural utterances. Consider the two alternative responses S2a and S2b in the following example:

U1: Hello, I would like to book a flight to London.
S2a: Which date did you have in mind?
S2b: Ok, flying to London on what date?

Whereas S2a only asks for the next piece of information to book the flight (uni-dimensional), S2b also gives feedback about the arrival city, allowing the user to correct any recognition errors (multi-dimensional). We aim to develop a principled multidimensional modelling and learning framework that covers a wide range of different phenomena, including the implicit confirmation in S2b.

This multi-dimensional approach will not only allow us to build systems that support more natural and effective interactions with users, but also enables cost-effective development of such interfaces for a variety of domains by learning transferable conversational skills (e.g., selecting actions in domain independent dimensions). We will therefore demonstrate our approach by building interactive spoken language interfaces for multiple application domains in a home automation scenario, allowing users to interact with for example their Smart TV or heating control system. We will closely collaborate with the industrial partner SemVox to explore this scenario.

The project will bring together expertise in statistical machine learning approaches to state-of-the-art spoken dialogue systems and natural language generation, as well as linguistic theories of multi-dimensional dialogue modelling (collaborating in particular with academic partner Prof. Bunt). MaDrIgAL will develop Next Generation Interaction Technologies relevant to Health Technology and Assisted Living, as well as tackle the question of a common user interface to the Internet of Things and Big Data.

From an end-user importance point of view, this work will be of interest to a wide range of businesses and companies in the UK, operating in areas such as speech technology, AI, home automation, etcetera. Specifically, we will develop and release a new spoken dialogue system (SDS) architecture compliant with the recently developed ISO standard for dialogue act annotation. This will promote interoperability, enabling the developer community to collaborate and reuse each other's components more easily, and ultimately help industry in developing products with spoken language interfaces more efficiently. The proposed research will also help to strengthen the impact of statistical SDS. Development costs for statistical techniques seem currently too high for manufacturers, due to the requirement of sufficient domain-specific data. The proposed new modular architecture indirectly addresses the economic feasibility of developing products with interactive interfaces for a variety of application domains.

This will ultimately help the UK to push past the US, which is still the main competitor in this area, despite the emergence of new language technology companies such as VocalIQ and Arria. For example, a senior speech scientist at Apple has said publicly that statistical methods will form the algorithmic basis for Siri in the near future. These increased capabilities will also enable us to tackle more complex interaction scenarios, such as social robotics, situated multimodal dialogue with smart devices, and eventually controlling and managing the Internet of Things.

From a societal importance point of view, this project will create a unified user interface which allows every-day users to access and control the Internet of Things in an intuitive way using natural language, and as such lowers the barrier to access and benefit from new technology. This is especially relevant for elderly and/or disabled users in a home automation scenario for example.