Future Everyday Interaction with the Autonomous Internet of Things

Lead Research Organisation: University of Southampton
Department Name: Electronics and Computer Science

Abstract

This project seeks to investigate the design of interaction mechanisms and user interfaces for a future Autonomous Internet of Things (A-IoT): a system of interconnected devices that reaches beyond most current incarnations of the IoT to include aspects of autonomy or automation as a key feature. Nascent instantiations of the A-IoT range from smart thermostats that learn to autonomously control central heating systems based on the presence of users and their routine, to washing machines that order detergent for delivery when it runs out. In other words, this A-IoT can proactively respond to sensed environmental changes, effectively doing work on behalf of users, with the promise of a more efficient use of resources (e.g. to use less energy for heating) or increased convenience (e.g. to always have detergent available).
The wealth (or "deluge") of data produced by the IoT is likely to keep growing beyond human capacity to turn it into meaningful information that can be acted on. Therefore, it will require future interactive systems to increasingly support the delegation of granular decision making over large and complex data to autonomous computational agents, allowing users to make informed choices about their general needs and comfort. In an Autonomous IoT; data and decisions will be, in part, 'actively' managed by the devices and their software, drawing upon machine learning techniques and optimization algorithms.
However, recent studies examining the real-world acceptance of a commercial smart thermostat highlighted how errors, limited legibility of the system operation, and excessive user expectations caused frustration and led to some users abandoning the technology. Our own prior work revealed people distrust a potential smart energy infrastructure due to lack of accountability of the ownership, intent, and permitted activities of the autonomous technology. These results suggest that the design of A-IoT systems needs to address several challenges to be made accountable; including, on the system side, designing autonomous decision-making to take into account the uncertain nature of contingent human behaviour; and on the user side, the need to make these systems legible and usable in everyday life. Indeed there is an inherent tension between making a system's operation legible and not overwhelming users with the technical complexity of artificial intelligence algorithms. To date, the methodologies to design such systems are rather sparse and not specific to A-IoT systems (spanning HCI, AI, and Ubicomp) and hence a more focused approach is required to determine the core design principles and methods for the implementation of A-IoT systems.
Our goal is thus to establish the scientific underpinnings of user interactions with A-IoT systems, in a domestic everyday context, with the aim of elucidate the following research questions: to what extent may users be willing to delegate agency to A-IoT systems in everyday contexts? How should interactions with A-IoT systems be engineered to support rather than hinder users' daily activities? What capabilities are essential for intelligent agents to manage such A-IoT systems? How can we design such systems so that they allow users to delegate control, yet easily regain it? Unless such questions are fully addressed, A-IOT systems are likely to frustrate users, resulting in significant waste of time and resources.
Hence, we will address these challenges through a combination of techniques, including the study of existing practices, the iterative development of novel A-IoT prototypes and their evaluation in-the-wild. Such a multidisciplinary approach is made possible by a team that brings together internationally-leading researchers in human-computer interaction, artificial intelligence and design ethnography.

Planned Impact

The Internet of Things has been identified by the UK government as a key area for investment, recognizing its high potential for impact on the national economy and, more in general, upon society (www.gov.uk/government/publications/internet-of-things-blackett-review). Equally, autonomous systems have been recognized by the EPSRC as a priority area and "part of [their] response to national challenges". This project addresses both these recognized innovation opportunities through its aim to combine IoT and autonomous intelligent systems into the A-IoT and release its potential for applications in domestic everyday settings.

The proposed research cuts across three of the six priority research areas set out in the roadmap for interdisciplinary research on the Internet of Things; namely People, Trust, and Data [IoTSIG 2013].

Our focus on domestic practices relates to a broad range of activities such as supply (production and distribution), storage, food preparation, eating and waste reduction, a central societal concern in the UK. These cut across a range of key societal sectors including agriculture, manufacturing, transportation, gastronomy and energy. The results of the proposed research will be relevant to stakeholders from these sectors, informing how their activities might be supported by the A-IoT.

End-users engagement
The approach through which the project aims are achieved is an inclusive, user-centred design process, involving end-users at all stages of the design process. Participatory design and envisionment workshops, and field deployments of prototypes will take place throughout the project. These activities will involve our project partners, specific food consumer groups and commercial food venues (reached through our partners), as well as members of the general public.

Industry engagement
Industry engagement is undertaken with and through our project partners: Wireless Things PLC and Senseye, as IoT technology providers, Sutton Community Farm and Homemade Cafe Ltd. as potential IoT technology beneficiaries. The partners will shape the design of our prototypes and trials, in order to make our results relevant to their needs. We will also showcase achievements and outputs, including new IoT application and services, at a suitable industry events (e.g. Innovate UK conferences), to engage other key industry players in the electronic technology, UX design and food supply sectors. We will allocate the role of managing and furthering interactions with industrial partners to Ramchurn, given his experience running a number of successful Knowledge Transfer Secondment activities with Hampshire County Council and BAE systems as well as running the Industrial Placements programme for the Electronics and Computer Science department at Southampton.

Communication & Press activity
An advisory panel of stakeholders will be convened, consisting of industry, the third sector, governing bodies, and external academics especially from fields not included in the project team. We will hold annual all-hand meetings with the panel to present and reflect research progress, and to seek strategic research guidance from stakeholders, specifically in relation to impact performance, and further opportunities to disseminate findings.

Project output public release
In addition to the normal academic dissemination routes the research findings will be made available through summaries and briefing papers, including a final report detailing the project aims and key headline descriptive results. These documents will be made available on a dedicated project website. To maximise impact we will host a dissemination event at the end of the programme.
This approach will follow a similar format to the 'outcomes' section of other EPSRC projects from the applicants (e.g. ORCHID, HORIZON), which resulted in national and international media coverage (e.g. BBC, Guardian, Independent, New Scientist, C4).

Publications

10 25 50
publication icon
Nilsson T (2016) Applying Seamful Design in Location-Based Mobile Museum Applications in ACM Transactions on Multimedia Computing, Communications, and Applications

publication icon
Alan A (2016) Tariff Agent Interacting with a Future Smart Energy System at Home in ACM Transactions on Computer-Human Interaction

publication icon
Porcheron M (2017) "Do Animals Have Accents?"

publication icon
Fischer JE (2017) Data Work: How Energy Advisors and Clients Make IoT Data Accountable. in Computer supported cooperative work : CSCW : an international journal

publication icon
Fischer J (2017) In-the-loop or on-the-loop? Interactional arrangements to support team coordination with a planning agent in Concurrency and Computation: Practice and Experience

publication icon
Kwon H (2018) The Connected Shower Studying Intimate Data in Everyday Life in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

publication icon
Verame J (2018) Learning from the Veg Box

 
Description Please refer to the entry for EP/N014243/2
Exploitation Route Please refer to the entry for EP/N014243/2
Sectors Agriculture

Food and Drink

Digital/Communication/Information Technologies (including Software)

Other

 
Description DFID Transform
Amount £299,500 (GBP)
Funding ID RG0970 
Organisation UK Department for International Development 
Sector Public
Country United Kingdom
Start 09/2018 
End 09/2020
 
Description EPSRC CASE studentship with Microsoft Research
Amount £116,200 (GBP)
Funding ID CASE voucher 18000109 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 09/2018 
End 09/2022
 
Description Platform Grants
Amount £1,431,420 (GBP)
Funding ID EP/P010164/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 03/2017 
End 04/2022
 
Description Smart Products Beacon Demonstrator project
Amount £141,000 (GBP)
Funding ID RIS 657510 
Organisation University of Nottingham 
Sector Academic/University
Country United Kingdom
Start 11/2018 
End 04/2020
 
Title Chariot 2.0 for domestic energy monitoring 
Description CharIoT is a web platform that aims to leverage the "Internet of Things" to support people in managing their energy use particularly those affected by fuel poverty. The platform relies on a sensor kit to capture environmental data that is meaning in the context of fuel poverty (such as humidity and temperature), and explore ways in which to interrogate the data for the 'symptoms' of fuel poverty, such as damp and cold. Chariot is a collaboration between researchers at the University of Nottingham, the University of Southampton, and the Centre for Sustainable Energy. This project was funded by the Engineering and Physical Sciences Research Council (EPSRC). 
Type Of Technology Webtool/Application 
Year Produced 2017 
Open Source License? Yes  
Impact The Centre for Sustainable Energy is using it actively in a energy monitoring project. 
URL http://chariot.org.uk/
 
Title NottReal: A Tool for Voice-based Wizard of Oz studies 
Description NottReal is an application designed for simulating Voice User Interfaces (VUIs) in Wizard of Oz studies. 
Type Of Technology Software 
Year Produced 2020 
Open Source License? Yes  
Impact This has underpinned a number of research studies leading to several publications 
URL https://github.com/MixedRealityLab/nottreal/
 
Description BBC Studio Day 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Media (as a channel to the public)
Results and Impact Talk and workshop with staff at BBC teams relating to design and user experience, based on voice-based interfaces. Disseminating the findings relating to this award to industry practitioners.
Year(s) Of Engagement Activity 2017
 
Description BBC User Experience and Design 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Industry/Business
Results and Impact Talk BBC User Experience & Design team
Year(s) Of Engagement Activity 2017
 
Description CHI Lites talk "The trouble with voice interfaces", April 2018, Montre´al, Canada. 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact An invited talk to a public audience collocated with the premier conference in my field of research.
Year(s) Of Engagement Activity 2018
URL https://youtu.be/Gg8VU9NJqRU
 
Description Conversational interaction and Conversation Analysis 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Invited talk to industry researchers at Microsoft Research Cambridge.
Year(s) Of Engagement Activity 2017
 
Description Invited panellist on Innovations in Autonomous Systems 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Panel at ACM CSCW 2016.
Year(s) Of Engagement Activity 2016
URL https://dl.acm.org/citation.cfm?id=2818052.2893361
 
Description Panel at Centrica 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Industry/Business
Results and Impact Expert panel about AI and HCI in the context of energy at an internal event at Centrica in Windsor
Year(s) Of Engagement Activity 2017
 
Description Pint of Science Talk 2017 - Conversations with machines: pipe dream or the next big thing? 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Public/other audiences
Results and Impact Conversations with machines: pipe dream or the next big thing?

Natural language interfaces have recently made it into popular commercial products from smartphones to standalone devices such as the Amazon Echo. In this talk I will use examples from our research in which we examine how people talk to these devices in real life. Yet, the broken interactions we have recorded can hardly be described as 'conversation'. I will discuss why it is so hard to build conversational interfaces and share some ideas of what we would need to do to change this.
Year(s) Of Engagement Activity 2017
URL https://pintofscience.co.uk/events/nottingham
 
Description Smarter Warmer Homes 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Third sector organisations
Results and Impact Free workshop: 11 Oct, Bristol - How can smart energy tech (meters, data, sensors etc) help vulnerable people?
Year(s) Of Engagement Activity 2017
URL http://www.bristolhealthpartners.org.uk/events/view/2017/10/11/smarter-warmer-homes-realising-benefi...
 
Description The trouble with Alexa: voice interfaces and conversation design. Keynote at SICSA Workshop on Conversational AI. Heriot-Watt University. Edinburgh. 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Postgraduate students
Results and Impact An invited keynote talk at a workshop with primary academic, but also industry practitioners
Year(s) Of Engagement Activity 2018
URL https://sites.google.com/site/workshoponconversationalai/keynotes