Future Everyday Interaction with the Autonomous Internet of Things
Lead Research Organisation:
University of Southampton
Department Name: Electronics and Computer Science
Abstract
This project seeks to investigate the design of interaction mechanisms and user interfaces for a future Autonomous Internet of Things (A-IoT): a system of interconnected devices that reaches beyond most current incarnations of the IoT to include aspects of autonomy or automation as a key feature. Nascent instantiations of the A-IoT range from smart thermostats that learn to autonomously control central heating systems based on the presence of users and their routine, to washing machines that order detergent for delivery when it runs out. In other words, this A-IoT can proactively respond to sensed environmental changes, effectively doing work on behalf of users, with the promise of a more efficient use of resources (e.g. to use less energy for heating) or increased convenience (e.g. to always have detergent available).
The wealth (or "deluge") of data produced by the IoT is likely to keep growing beyond human capacity to turn it into meaningful information that can be acted on. Therefore, it will require future interactive systems to increasingly support the delegation of granular decision making over large and complex data to autonomous computational agents, allowing users to make informed choices about their general needs and comfort. In an Autonomous IoT; data and decisions will be, in part, 'actively' managed by the devices and their software, drawing upon machine learning techniques and optimization algorithms.
However, recent studies examining the real-world acceptance of a commercial smart thermostat highlighted how errors, limited legibility of the system operation, and excessive user expectations caused frustration and led to some users abandoning the technology. Our own prior work revealed people distrust a potential smart energy infrastructure due to lack of accountability of the ownership, intent, and permitted activities of the autonomous technology. These results suggest that the design of A-IoT systems needs to address several challenges to be made accountable; including, on the system side, designing autonomous decision-making to take into account the uncertain nature of contingent human behaviour; and on the user side, the need to make these systems legible and usable in everyday life. Indeed there is an inherent tension between making a system's operation legible and not overwhelming users with the technical complexity of artificial intelligence algorithms. To date, the methodologies to design such systems are rather sparse and not specific to A-IoT systems (spanning HCI, AI, and Ubicomp) and hence a more focused approach is required to determine the core design principles and methods for the implementation of A-IoT systems.
Our goal is thus to establish the scientific underpinnings of user interactions with A-IoT systems, in a domestic everyday context, with the aim of elucidate the following research questions: to what extent may users be willing to delegate agency to A-IoT systems in everyday contexts? How should interactions with A-IoT systems be engineered to support rather than hinder users' daily activities? What capabilities are essential for intelligent agents to manage such A-IoT systems? How can we design such systems so that they allow users to delegate control, yet easily regain it? Unless such questions are fully addressed, A-IOT systems are likely to frustrate users, resulting in significant waste of time and resources.
Hence, we will address these challenges through a combination of techniques, including the study of existing practices, the iterative development of novel A-IoT prototypes and their evaluation in-the-wild. Such a multidisciplinary approach is made possible by a team that brings together internationally-leading researchers in human-computer interaction, artificial intelligence and design ethnography.
The wealth (or "deluge") of data produced by the IoT is likely to keep growing beyond human capacity to turn it into meaningful information that can be acted on. Therefore, it will require future interactive systems to increasingly support the delegation of granular decision making over large and complex data to autonomous computational agents, allowing users to make informed choices about their general needs and comfort. In an Autonomous IoT; data and decisions will be, in part, 'actively' managed by the devices and their software, drawing upon machine learning techniques and optimization algorithms.
However, recent studies examining the real-world acceptance of a commercial smart thermostat highlighted how errors, limited legibility of the system operation, and excessive user expectations caused frustration and led to some users abandoning the technology. Our own prior work revealed people distrust a potential smart energy infrastructure due to lack of accountability of the ownership, intent, and permitted activities of the autonomous technology. These results suggest that the design of A-IoT systems needs to address several challenges to be made accountable; including, on the system side, designing autonomous decision-making to take into account the uncertain nature of contingent human behaviour; and on the user side, the need to make these systems legible and usable in everyday life. Indeed there is an inherent tension between making a system's operation legible and not overwhelming users with the technical complexity of artificial intelligence algorithms. To date, the methodologies to design such systems are rather sparse and not specific to A-IoT systems (spanning HCI, AI, and Ubicomp) and hence a more focused approach is required to determine the core design principles and methods for the implementation of A-IoT systems.
Our goal is thus to establish the scientific underpinnings of user interactions with A-IoT systems, in a domestic everyday context, with the aim of elucidate the following research questions: to what extent may users be willing to delegate agency to A-IoT systems in everyday contexts? How should interactions with A-IoT systems be engineered to support rather than hinder users' daily activities? What capabilities are essential for intelligent agents to manage such A-IoT systems? How can we design such systems so that they allow users to delegate control, yet easily regain it? Unless such questions are fully addressed, A-IOT systems are likely to frustrate users, resulting in significant waste of time and resources.
Hence, we will address these challenges through a combination of techniques, including the study of existing practices, the iterative development of novel A-IoT prototypes and their evaluation in-the-wild. Such a multidisciplinary approach is made possible by a team that brings together internationally-leading researchers in human-computer interaction, artificial intelligence and design ethnography.
Planned Impact
The Internet of Things has been identified by the UK government as a key area for investment, recognizing its high potential for impact on the national economy and, more in general, upon society (www.gov.uk/government/publications/internet-of-things-blackett-review). Equally, autonomous systems have been recognized by the EPSRC as a priority area and "part of [their] response to national challenges". This project addresses both these recognized innovation opportunities through its aim to combine IoT and autonomous intelligent systems into the A-IoT and release its potential for applications in domestic everyday settings.
The proposed research cuts across three of the six priority research areas set out in the roadmap for interdisciplinary research on the Internet of Things; namely People, Trust, and Data [IoTSIG 2013].
Our focus on domestic practices relates to a broad range of activities such as supply (production and distribution), storage, food preparation, eating and waste reduction, a central societal concern in the UK. These cut across a range of key societal sectors including agriculture, manufacturing, transportation, gastronomy and energy. The results of the proposed research will be relevant to stakeholders from these sectors, informing how their activities might be supported by the A-IoT.
End-users engagement
The approach through which the project aims are achieved is an inclusive, user-centred design process, involving end-users at all stages of the design process. Participatory design and envisionment workshops, and field deployments of prototypes will take place throughout the project. These activities will involve our project partners, specific food consumer groups and commercial food venues (reached through our partners), as well as members of the general public.
Industry engagement
Industry engagement is undertaken with and through our project partners: Wireless Things PLC and Senseye, as IoT technology providers, Sutton Community Farm and Homemade Cafe Ltd. as potential IoT technology beneficiaries. The partners will shape the design of our prototypes and trials, in order to make our results relevant to their needs. We will also showcase achievements and outputs, including new IoT application and services, at a suitable industry events (e.g. Innovate UK conferences), to engage other key industry players in the electronic technology, UX design and food supply sectors. We will allocate the role of managing and furthering interactions with industrial partners to Ramchurn, given his experience running a number of successful Knowledge Transfer Secondment activities with Hampshire County Council and BAE systems as well as running the Industrial Placements programme for the Electronics and Computer Science department at Southampton.
Communication & Press activity
An advisory panel of stakeholders will be convened, consisting of industry, the third sector, governing bodies, and external academics especially from fields not included in the project team. We will hold annual all-hand meetings with the panel to present and reflect research progress, and to seek strategic research guidance from stakeholders, specifically in relation to impact performance, and further opportunities to disseminate findings.
Project output public release
In addition to the normal academic dissemination routes the research findings will be made available through summaries and briefing papers, including a final report detailing the project aims and key headline descriptive results. These documents will be made available on a dedicated project website. To maximise impact we will host a dissemination event at the end of the programme.
This approach will follow a similar format to the 'outcomes' section of other EPSRC projects from the applicants (e.g. ORCHID, HORIZON), which resulted in national and international media coverage (e.g. BBC, Guardian, Independent, New Scientist, C4).
The proposed research cuts across three of the six priority research areas set out in the roadmap for interdisciplinary research on the Internet of Things; namely People, Trust, and Data [IoTSIG 2013].
Our focus on domestic practices relates to a broad range of activities such as supply (production and distribution), storage, food preparation, eating and waste reduction, a central societal concern in the UK. These cut across a range of key societal sectors including agriculture, manufacturing, transportation, gastronomy and energy. The results of the proposed research will be relevant to stakeholders from these sectors, informing how their activities might be supported by the A-IoT.
End-users engagement
The approach through which the project aims are achieved is an inclusive, user-centred design process, involving end-users at all stages of the design process. Participatory design and envisionment workshops, and field deployments of prototypes will take place throughout the project. These activities will involve our project partners, specific food consumer groups and commercial food venues (reached through our partners), as well as members of the general public.
Industry engagement
Industry engagement is undertaken with and through our project partners: Wireless Things PLC and Senseye, as IoT technology providers, Sutton Community Farm and Homemade Cafe Ltd. as potential IoT technology beneficiaries. The partners will shape the design of our prototypes and trials, in order to make our results relevant to their needs. We will also showcase achievements and outputs, including new IoT application and services, at a suitable industry events (e.g. Innovate UK conferences), to engage other key industry players in the electronic technology, UX design and food supply sectors. We will allocate the role of managing and furthering interactions with industrial partners to Ramchurn, given his experience running a number of successful Knowledge Transfer Secondment activities with Hampshire County Council and BAE systems as well as running the Industrial Placements programme for the Electronics and Computer Science department at Southampton.
Communication & Press activity
An advisory panel of stakeholders will be convened, consisting of industry, the third sector, governing bodies, and external academics especially from fields not included in the project team. We will hold annual all-hand meetings with the panel to present and reflect research progress, and to seek strategic research guidance from stakeholders, specifically in relation to impact performance, and further opportunities to disseminate findings.
Project output public release
In addition to the normal academic dissemination routes the research findings will be made available through summaries and briefing papers, including a final report detailing the project aims and key headline descriptive results. These documents will be made available on a dedicated project website. To maximise impact we will host a dissemination event at the end of the programme.
This approach will follow a similar format to the 'outcomes' section of other EPSRC projects from the applicants (e.g. ORCHID, HORIZON), which resulted in national and international media coverage (e.g. BBC, Guardian, Independent, New Scientist, C4).
Publications
Fischer J
(2016)
New horizons for the IoT in everyday life
Fischer J
(2016)
"Just whack it on until it gets hot"
Alan A
(2016)
Tariff Agent Interacting with a Future Smart Energy System at Home
in ACM Transactions on Computer-Human Interaction
Nilsson T
(2016)
Applying Seamful Design in Location-Based Mobile Museum Applications
in ACM Transactions on Multimedia Computing, Communications, and Applications
Alan AT
(2016)
It is too hot: an in-situ study of three designs for heating
Fischer, J
(2016)
"Just whack it on until it gets hot": Working with IoT Data in the Home
Fischer J
(2017)
In-the-loop or on-the-loop? Interactional arrangements to support team coordination with a planning agent
in Concurrency and Computation: Practice and Experience
Porcheron M
(2017)
"Do Animals Have Accents?"
Porcheron M
(2017)
Talking with Conversational Agents in Collaborative Action
Fischer JE
(2017)
Data Work: How Energy Advisors and Clients Make IoT Data Accountable.
in Computer supported cooperative work : CSCW : an international journal
Fischer J
(2018)
Beyond "Same Time, Same Place": Introduction to the Special Issue on Collocated Interaction
in Human-Computer Interaction
Hyland L
(2018)
"What do you want for dinner?" - need anticipation and the design of proactive technologies for the home
in Computer Supported Cooperative Work (CSCW)
Reeves S
(2018)
'This is not what we wanted' designing for conversation with voice interfaces
in Interactions
García García P
(2018)
Seeing (Movement) is Believing: The Effect of Motion on Perception of Automatic Systems Performance
in Human-Computer Interaction
Kwon H
(2018)
The Connected Shower Studying Intimate Data in Everyday Life
in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Verame J
(2018)
Learning from the Veg Box
Porcheron M
(2018)
Voice Interfaces in Everyday Life
Bicego M
(2018)
On the distinctiveness of the electricity load profile
in Pattern Recognition
Costanza E
(2018)
Digital Technology and Sustainability Engaging the Paradox
Liu C
(2019)
Rigor, Relevance and Impact
Kittley-Davies J
(2019)
Evaluating the Effect of Feedback from Different Computer Vision Processing Stages
Fuentes C
(2019)
New Directions for the IoT
Fischer J
(2019)
Progressivity for voice interface design
Nilsson T
(2019)
Creating Worlds
Nilsson T
(2019)
Breaching the future: understanding human challenges of autonomous systems for the home
in Personal and Ubiquitous Computing
Crabtree A
(2019)
Probing IoT-based consumer services: 'insights' from the connected shower
in Personal and Ubiquitous Computing
Fuentes C
(2019)
Tracking the Consumption of Home Essentials
Nilsson T
(2020)
Visions, Values, and Videos
Porcheron M
(2020)
CUI@CSCW: Collaborating through Conversational User Interfaces
Kirschthaler P
(2020)
What Can I Say?
Porcheron M
(2020)
NottReal
Castle-Green T
(2020)
Decision Trees as Sociotechnical Objects in Chatbot Design
Alqaraawi A
(2020)
Evaluating saliency map explanations for convolutional neural networks
Herrmann M
(2021)
Exploring domestic energy consumption feedback through interactive annotation
in Energy Efficiency
Boudouraki A
(2021)
"I can't get round" Recruiting Assistance in Mobile Robotic Telepresence
in Proceedings of the ACM on Human-Computer Interaction
Porcheron M
(2021)
Pulling Back the Curtain on the Wizards of Oz
in Proceedings of the ACM on Human-Computer Interaction
| Description | Please refer to the entry for EP/N014243/2 |
| Exploitation Route | Please refer to the entry for EP/N014243/2 |
| Sectors | Agriculture, Food and Drink,Digital/Communication/Information Technologies (including Software),Other |
| Description | DFID Transform |
| Amount | £299,500 (GBP) |
| Funding ID | RG0970 |
| Organisation | UK Department for International Development |
| Sector | Public |
| Country | United Kingdom |
| Start | 10/2018 |
| End | 09/2020 |
| Description | EPSRC CASE studentship with Microsoft Research |
| Amount | £116,200 (GBP) |
| Funding ID | CASE voucher 18000109 |
| Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
| Sector | Public |
| Country | United Kingdom |
| Start | 10/2018 |
| End | 09/2022 |
| Description | Platform Grants |
| Amount | £1,431,420 (GBP) |
| Funding ID | EP/P010164/1 |
| Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
| Sector | Public |
| Country | United Kingdom |
| Start | 04/2017 |
| End | 04/2022 |
| Description | Smart Products Beacon Demonstrator project |
| Amount | £141,000 (GBP) |
| Funding ID | RIS 657510 |
| Organisation | University of Nottingham |
| Sector | Academic/University |
| Country | United Kingdom |
| Start | 11/2018 |
| End | 04/2020 |
| Title | Chariot 2.0 for domestic energy monitoring |
| Description | CharIoT is a web platform that aims to leverage the "Internet of Things" to support people in managing their energy use particularly those affected by fuel poverty. The platform relies on a sensor kit to capture environmental data that is meaning in the context of fuel poverty (such as humidity and temperature), and explore ways in which to interrogate the data for the 'symptoms' of fuel poverty, such as damp and cold. Chariot is a collaboration between researchers at the University of Nottingham, the University of Southampton, and the Centre for Sustainable Energy. This project was funded by the Engineering and Physical Sciences Research Council (EPSRC). |
| Type Of Technology | Webtool/Application |
| Year Produced | 2017 |
| Open Source License? | Yes |
| Impact | The Centre for Sustainable Energy is using it actively in a energy monitoring project. |
| URL | http://chariot.org.uk/ |
| Title | NottReal: A Tool for Voice-based Wizard of Oz studies |
| Description | NottReal is an application designed for simulating Voice User Interfaces (VUIs) in Wizard of Oz studies. |
| Type Of Technology | Software |
| Year Produced | 2020 |
| Open Source License? | Yes |
| Impact | This has underpinned a number of research studies leading to several publications |
| URL | https://github.com/MixedRealityLab/nottreal/ |
| Description | BBC Studio Day |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Media (as a channel to the public) |
| Results and Impact | Talk and workshop with staff at BBC teams relating to design and user experience, based on voice-based interfaces. Disseminating the findings relating to this award to industry practitioners. |
| Year(s) Of Engagement Activity | 2017 |
| Description | BBC User Experience and Design |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Industry/Business |
| Results and Impact | Talk BBC User Experience & Design team |
| Year(s) Of Engagement Activity | 2017 |
| Description | CHI Lites talk "The trouble with voice interfaces", April 2018, Montre´al, Canada. |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Public/other audiences |
| Results and Impact | An invited talk to a public audience collocated with the premier conference in my field of research. |
| Year(s) Of Engagement Activity | 2018 |
| URL | https://youtu.be/Gg8VU9NJqRU |
| Description | Conversational interaction and Conversation Analysis |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | Invited talk to industry researchers at Microsoft Research Cambridge. |
| Year(s) Of Engagement Activity | 2017 |
| Description | Invited panellist on Innovations in Autonomous Systems |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Panel at ACM CSCW 2016. |
| Year(s) Of Engagement Activity | 2016 |
| URL | https://dl.acm.org/citation.cfm?id=2818052.2893361 |
| Description | Panel at Centrica |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Industry/Business |
| Results and Impact | Expert panel about AI and HCI in the context of energy at an internal event at Centrica in Windsor |
| Year(s) Of Engagement Activity | 2017 |
| Description | Pint of Science Talk 2017 - Conversations with machines: pipe dream or the next big thing? |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Public/other audiences |
| Results and Impact | Conversations with machines: pipe dream or the next big thing? Natural language interfaces have recently made it into popular commercial products from smartphones to standalone devices such as the Amazon Echo. In this talk I will use examples from our research in which we examine how people talk to these devices in real life. Yet, the broken interactions we have recorded can hardly be described as 'conversation'. I will discuss why it is so hard to build conversational interfaces and share some ideas of what we would need to do to change this. |
| Year(s) Of Engagement Activity | 2017 |
| URL | https://pintofscience.co.uk/events/nottingham |
| Description | Smarter Warmer Homes |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Third sector organisations |
| Results and Impact | Free workshop: 11 Oct, Bristol - How can smart energy tech (meters, data, sensors etc) help vulnerable people? |
| Year(s) Of Engagement Activity | 2017 |
| URL | http://www.bristolhealthpartners.org.uk/events/view/2017/10/11/smarter-warmer-homes-realising-benefi... |
| Description | The trouble with Alexa: voice interfaces and conversation design. Keynote at SICSA Workshop on Conversational AI. Heriot-Watt University. Edinburgh. |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Postgraduate students |
| Results and Impact | An invited keynote talk at a workshop with primary academic, but also industry practitioners |
| Year(s) Of Engagement Activity | 2018 |
| URL | https://sites.google.com/site/workshoponconversationalai/keynotes |