Future Everyday Interaction with the Autonomous Internet of Things

Lead Research Organisation: University College London
Department Name: UCL Interaction Centre


This project seeks to investigate the design of interaction mechanisms and user interfaces for a future Autonomous Internet of Things (A-IoT): a system of interconnected devices that reaches beyond most current incarnations of the IoT to include aspects of autonomy or automation as a key feature. Nascent instantiations of the A-IoT range from smart thermostats that learn to autonomously control central heating systems based on the presence of users and their routine, to washing machines that order detergent for delivery when it runs out. In other words, this A-IoT can proactively respond to sensed environmental changes, effectively doing work on behalf of users, with the promise of a more efficient use of resources (e.g. to use less energy for heating) or increased convenience (e.g. to always have detergent available).
The wealth (or "deluge") of data produced by the IoT is likely to keep growing beyond human capacity to turn it into meaningful information that can be acted on. Therefore, it will require future interactive systems to increasingly support the delegation of granular decision making over large and complex data to autonomous computational agents, allowing users to make informed choices about their general needs and comfort. In an Autonomous IoT; data and decisions will be, in part, 'actively' managed by the devices and their software, drawing upon machine learning techniques and optimization algorithms.
However, recent studies examining the real-world acceptance of a commercial smart thermostat highlighted how errors, limited legibility of the system operation, and excessive user expectations caused frustration and led to some users abandoning the technology. Our own prior work revealed people distrust a potential smart energy infrastructure due to lack of accountability of the ownership, intent, and permitted activities of the autonomous technology. These results suggest that the design of A-IoT systems needs to address several challenges to be made accountable; including, on the system side, designing autonomous decision-making to take into account the uncertain nature of contingent human behaviour; and on the user side, the need to make these systems legible and usable in everyday life. Indeed there is an inherent tension between making a system's operation legible and not overwhelming users with the technical complexity of artificial intelligence algorithms. To date, the methodologies to design such systems are rather sparse and not specific to A-IoT systems (spanning HCI, AI, and Ubicomp) and hence a more focused approach is required to determine the core design principles and methods for the implementation of A-IoT systems.
Our goal is thus to establish the scientific underpinnings of user interactions with A-IoT systems, in a domestic everyday context, with the aim of elucidate the following research questions: to what extent may users be willing to delegate agency to A-IoT systems in everyday contexts? How should interactions with A-IoT systems be engineered to support rather than hinder users' daily activities? What capabilities are essential for intelligent agents to manage such A-IoT systems? How can we design such systems so that they allow users to delegate control, yet easily regain it? Unless such questions are fully addressed, A-IOT systems are likely to frustrate users, resulting in significant waste of time and resources.
Hence, we will address these challenges through a combination of techniques, including the study of existing practices, the iterative development of novel A-IoT prototypes and their evaluation in-the-wild. Such a multidisciplinary approach is made possible by a team that brings together internationally-leading researchers in human-computer interaction, artificial intelligence and design ethnography.

Planned Impact

The Internet of Things has been identified by the UK government as a key area for investment, recognizing its high potential for impact on the national economy and, more in general, upon society (www.gov.uk/government/publications/internet-of-things-blackett-review). Equally, autonomous systems have been recognized by the EPSRC as a priority area and "part of [their] response to national challenges". This project addresses both these recognized innovation opportunities through its aim to combine IoT and autonomous intelligent systems into the A-IoT and release its potential for applications in domestic everyday settings.

The proposed research cuts across three of the six priority research areas set out in the roadmap for interdisciplinary research on the Internet of Things; namely People, Trust, and Data [IoTSIG 2013].

Our focus on domestic practices relates to a broad range of activities such as supply (production and distribution), storage, food preparation, eating and waste reduction, a central societal concern in the UK. These cut across a range of key societal sectors including agriculture, manufacturing, transportation, gastronomy and energy. The results of the proposed research will be relevant to stakeholders from these sectors, informing how their activities might be supported by the A-IoT.

End-users engagement
The approach through which the project aims are achieved is an inclusive, user-centred design process, involving end-users at all stages of the design process. Participatory design and envisionment workshops, and field deployments of prototypes will take place throughout the project. These activities will involve our project partners, specific food consumer groups and commercial food venues (reached through our partners), as well as members of the general public.

Industry engagement
Industry engagement is undertaken with and through our project partners: Wireless Things PLC and Senseye, as IoT technology providers, Sutton Community Farm and Homemade Cafe Ltd. as potential IoT technology beneficiaries. The partners will shape the design of our prototypes and trials, in order to make our results relevant to their needs. We will also showcase achievements and outputs, including new IoT application and services, at a suitable industry events (e.g. Innovate UK conferences), to engage other key industry players in the electronic technology, UX design and food supply sectors. We will allocate the role of managing and furthering interactions with industrial partners to Ramchurn, given his experience running a number of successful Knowledge Transfer Secondment activities with Hampshire County Council and BAE systems as well as running the Industrial Placements programme for the Electronics and Computer Science department at Southampton.

Communication & Press activity
An advisory panel of stakeholders will be convened, consisting of industry, the third sector, governing bodies, and external academics especially from fields not included in the project team. We will hold annual all-hand meetings with the panel to present and reflect research progress, and to seek strategic research guidance from stakeholders, specifically in relation to impact performance, and further opportunities to disseminate findings.

Project output public release
In addition to the normal academic dissemination routes the research findings will be made available through summaries and briefing papers, including a final report detailing the project aims and key headline descriptive results. These documents will be made available on a dedicated project website. To maximise impact we will host a dissemination event at the end of the programme.
This approach will follow a similar format to the 'outcomes' section of other EPSRC projects from the applicants (e.g. ORCHID, HORIZON), which resulted in national and international media coverage (e.g. BBC, Guardian, Independent, New Scientist, C4).

Related Projects

Project Reference Relationship Related To Start End Award Value
EP/N014243/1 31/03/2016 20/09/2016 £806,241
EP/N014243/2 Transfer EP/N014243/1 21/12/2016 31/10/2019 £732,370
Description The Internet of Things (IoT) promises to enable applications that foster a more efficient, sustainable, and healthy way of life. If end-users are to take full advantage of these developments we foresee the need for future IoT systems and services to include an element of autonomy and support the delegation of agency to software processes and connected devices.

To inform the design of such future technology, this project involved mainly on two streams of investigation. In the first stream, field studies were conducted with households to investigate existing practices of agency delegation around food, and interaction with novel prototypes that our team designed and built to monitor or mediate food shopping and food waste. The second stream focussed instead on more controlled studies, aimed at investigating the effects of specific design features of Autonomous IoT systems can influence users perception of them.

Key findings from across bot streams include:

- Users are able to take advantage of agency delegation, even when the system involves considerable uncertainty; however, agency delegation must be warranted, it must be possible to incorporate delegated decisions into everyday activities, and delegation is subject to constraint. Design implications of this work include that autonomous IoT systems need to support people's diverse values, and their coordinative and creative practices.

- The 'cycle' of household goods in the home is shaped by the underlying contingencies of everyday life. Our analysis has identified eight categories of contingencies participants used frequently to explain why items were used up earlier or later than predicted by a simple linear algorithm; these categories were routine changes (e.g. holidays), sporadic events (e.g. guests visiting), preferences (e.g. enjoyment), location (e.g. forgotten about), normal use (e.g. batch cooking), probe (e.g. forgot to scan), product quality (e.g. different brand), freshness (e.g. frozen). We also found whether products are seen as equivalent to others is contingent a range of factors including product class, where it was bought, and a host of product attributes. A take-away is that these contingencies drive the variability that is part and parcel of everyday product use and thus need considering in the design of Autonomous IoT systems and services.

- A series of controlled studies revealed that the use of movement and animations around autonomous IoT devices and their interfaces can influence users' perception these systems' performance. We designed and conducted follow up studies to try and identify an explanation for this effect. Our results suggest that this effect is related to the users' mental model of the smart system. Indeed, our results suggest that displaying a high detail of animations that match people's mental model can influence people's perception of the performance of two smart systems that we tested. We were also able to show that this effect holds, even when these systems have a minimal decrease in their performance. While this modality has the potential to improve users' ratings of a smart system's performance, it may also pose the danger of making a system appear to work better than it actually does.

- In the context of an exploration of how everyday object may be transformed by the autonomous IoT, we focused our attentions on one prevalent desktop object, an interactive lamp, with three actuated joints, which allowed us to explore the interaction space of such devices. Our qualitative findings revealed that users' expectations of smart objects around them that are proactively behaving were gathered, and we showed that a quasi-social behaviour, politeness, and a learning curve in understanding each other are desirable. We further found that people have socialised understandings of mechanical movement and therefore interactive technology needs to include knowledge about social norms.

These findings refer also to EP/N014243/1
Exploitation Route Please see implications for design of autonomous internet of things technology described together with the findings above

See also EP/N014243/1
Sectors Agriculture, Food and Drink,Digital/Communication/Information Technologies (including Software),Other

Description Beyond Individual Persuasion: Towards a Paradigm Shift in Interactive Visualisation and Sensing for Environmental Change
Amount £832,440 (GBP)
Funding ID EP/V042327/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 09/2021 
End 03/2024
Description Joint funded by the EPSRC IAA (impact accelerator account) at the University of Nottingham and Horizon Digital Economy Research
Amount £50,722 (GBP)
Organisation HORIZON Digital Economy Research 
Sector Academic/University
Country United Kingdom
Start 06/2016 
End 06/2017
Description Platform Grants
Amount £1,431,420 (GBP)
Funding ID EP/P010164/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 03/2017 
End 04/2022
Description Panel at Centrica 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Industry/Business
Results and Impact Expert panel about AI and HCI in the context of energy at an internal event at Centrica in Windsor
Year(s) Of Engagement Activity 2017