IDInteraction: Capturing Indicative Usage Models in Software for Implicit Device Interaction

The IDInteraction project asks: can we exploit models of human behaviour to move away from direct, unambiguous user commands, towards seamless user-device interaction? It will investigate and develop the techniques to capture 'Indicative Usage Models (IUMs), behavioural patterns or cues that precede a particular activity, and translate these into software-based 'Indicative Usage Patterns' (IUPs), to drive interaction with an app. The project will focus on future television broadcasting, examining the extent to which it is possible to capture IUMs from device sensor and event data, and deploy these as IUPs to pull content (additional information or related activities) to a 'second screen companion app', which a viewer watches on a mobile device alongside a TV programme.

The core of the research - investigating the extent to which we can anticipate user commands and requests - will play an important role in shaping the way we use future technology. The Internet of Things is fast becoming a reality, but the appropriate paradigms for interacting with it are far from understood. A rapidly growing number of connected devices means a potentially vast increase in the types of technology which people must use successfully. Whilst a person may be prepared to learn how to use a new home automation system or car, expecting him or her to do this for every machine with which there is a cursory encounter is unrealistic. We have already started to understand that interactive experiences which take fundamental human perception and thought processes into account are more successful than those which require significant learning on the part of the user. Implicit Device Interaction, where devices automatically know what the user wants, before a command is issued, is the next step.

The scenario investigated - second screen viewing - is also particularly timely. Broadcasters are keen to exploit the creative potential of a context where people are watching television with a mobile device, not least because it is set to become the principle form of TV viewing in the next few years. At present, however, the scenario is not well understood: whilst some research has focused on social aspects of this situation, very little has examined the perceptual or behavioural aspects of second screen interaction. Current companion apps, designed to complement the main programme, push information to the mobile device at given points in time, and this change on the secondary screen, which occurs in peripheral vision, is potentially distracting. A situation where information flow intuitively stops and starts according to the location of the viewer's attention is a highly desirable goal.

IDInteraction is an ambitious research project, investigating an aspect of human behaviour that is crucial to the future development of implicit user interfaces, and has been planned to tackle the problem from end-to-end. By studying a slice of the research problem, from modelling behaviour to testing a new user interface, it will provide an overview of the theoretical and technical challenges that the development of seamless user-device interaction will entail, and flag key areas for further investigation. From the perspective of effective software development, the project entails a considerable degree of risk: IUMs may be difficult to capture and deploy, and seamless information provision may be challenging to implement in this context. From the perspective of building theory in this area, such results would still have significant value, however. Improving our understanding of the limits of implicit interaction is crucial to moving this important, emergent, field forward.

IDInteraction will create societal and economic impacts from two perspectives.

(1) Determining the potential of using device sensor and event data to anticipate and pre-empt user commands and requests, leading to a significant shift, and potentially a major improvement, in the way in which we interact with technology.

(2) Investigating the increasingly common but under-researched home entertainment activity of watching television with a 'second screen' mobile device, and offering a rich understanding of the ways in which the viewing experience can be improved, in particular through the seamless provision of information at the moment the user requires it.

The impact of (1) is potentially very wide; the impact of (2) will be felt primarily within the domain of television broadcasting.

Key impacts, as described in the RCUK Typology, are therefore as follows:

'Contributing towards wealth creation and economic development'
IDInteraction will impact on how we use devices in the future. The models developed will be based, in the first instance, on interactivity during second screen viewing, but the techniques used for both creating Indicative Usage Models, and mapping them to software-based Indicative Usage Patterns, will have potential relevance for any interactive device. In particular, IDInteraction will contribute to our understanding of how to develop successful user interfaces for the rapidly growing 'Internet of Things', through examining anticipatory interaction with multiple networked devices. The step-change in interaction promised by this research will enhance 'economic prosperity' in the UK by improving the user experience of interactive devices, and enabling people to access information or services more quickly (even half a second saved during an interaction will have a significant effect when considered over long periods of time and across multiple users).

'Shaping and enhancing the effectiveness of a Public Service Broadcaster.'
The models and software outputs that will result from the project have already been identified by the BBC to be of direct relevance to their mission of delivering distinctive and transformational services both to the UK and internationally. The systematic investigation of second screen viewing entailed by IDInteraction and the resulting ability to provide content precisely when the viewer requires it, will both improve the user experience of watching television, and open the door to gathering a more detailed understanding of how people watch television with a mobile device.

'Enhancing the research capacity, knowledge and skills of a Public Service Broadcaster.'
The Department for Culture, Media and Sport have recently identified the production of 'world-beating, innovative content and services that originate here in the UK, but that are in demand across the globe,' as a strategic priority. UK media organisations, who have always been world leaders in broadcast technology, are finding it increasingly challenging to compete on the global stage (Future of Innovation in Television Technology Task Force Report, 2014). This research is therefore particularly timely from the perspective of supporting the UK television industry.

'Enhancing cultural enrichment and quality of life.'
The improvement to the viewing experience offered by the user-focused novel interaction paradigms investigated in IDInteration will also contribute to the Digital Economy 'Communities and Culture' theme. Watching television whilst using a mobile device is considered to be the dominant viewing environment of the future, and supporting it as effectively as possible will enhance education, enjoyment, relaxation and well-being.