Building an End-User Programming Framework for the Internet-of-Things
Lead Research Organisation:
University of Manchester
Department Name: Computer Science
Abstract
Description
The emerging infrastructure of the Internet-of-Things (IoT) offers unprecedented opportunities to build systems which integrate networks of things, services, and people. The expansion of the borders of systems to capture data from ambient and personal sensors, communicative tools, mobile and ubiquitous computing devices, defines a dramatic increase of the sensing substrate which can be used as the basis for the construction of the next generation of computing systems.
However, the heterogeneous and distributed nature of this substrate introduces new requirements for developing systems over the Internet-of-Things. Designing, programming or managing these new environments requires metaphors, interfaces, and interaction strategies for supporting programmers in the construction of IoT systems. Additionally, many IoT application scenarios require higher personalization and contextualization. Despite the sophistication and prevalence of user centered design methods, it is increasingly problematic for software developers to anticipate the needs of users at design time. In many situations, end-users themselves (either domain experts or consumers) should be enabled create and adapt systems to their preferences.
Approach
This project aims at building an end-to-end software development environment for the Internet-of-Things to support end-users in programming tasks. A central goal to the project is to enable end-users to build IoT applications using the combination of natural language commands, semantic search and visualization. The proposed programming environment is built upon the concept of schema-agnostic data environments, where end-users are abstracted away from the underlying data representation and can program using their own vocabulary.
Research Questions
1. Can semantic parsing (using Q-learning methods) provide the foundation for developing an end-user programming (EUP) environment using natural language for the Internet-of-Things?
2. How to support the evaluation of the quality of an end-user programming framework (building the supporting metrics and benchmarks)?
EPSRC Research Areas
Natural Language Processing, Pervasive and Ubiquitous Computing, Human-Computer Interaction
The emerging infrastructure of the Internet-of-Things (IoT) offers unprecedented opportunities to build systems which integrate networks of things, services, and people. The expansion of the borders of systems to capture data from ambient and personal sensors, communicative tools, mobile and ubiquitous computing devices, defines a dramatic increase of the sensing substrate which can be used as the basis for the construction of the next generation of computing systems.
However, the heterogeneous and distributed nature of this substrate introduces new requirements for developing systems over the Internet-of-Things. Designing, programming or managing these new environments requires metaphors, interfaces, and interaction strategies for supporting programmers in the construction of IoT systems. Additionally, many IoT application scenarios require higher personalization and contextualization. Despite the sophistication and prevalence of user centered design methods, it is increasingly problematic for software developers to anticipate the needs of users at design time. In many situations, end-users themselves (either domain experts or consumers) should be enabled create and adapt systems to their preferences.
Approach
This project aims at building an end-to-end software development environment for the Internet-of-Things to support end-users in programming tasks. A central goal to the project is to enable end-users to build IoT applications using the combination of natural language commands, semantic search and visualization. The proposed programming environment is built upon the concept of schema-agnostic data environments, where end-users are abstracted away from the underlying data representation and can program using their own vocabulary.
Research Questions
1. Can semantic parsing (using Q-learning methods) provide the foundation for developing an end-user programming (EUP) environment using natural language for the Internet-of-Things?
2. How to support the evaluation of the quality of an end-user programming framework (building the supporting metrics and benchmarks)?
EPSRC Research Areas
Natural Language Processing, Pervasive and Ubiquitous Computing, Human-Computer Interaction
People |
ORCID iD |
| Andre Freitas (Primary Supervisor) | |
| Filippos Ventirozos (Student) |
Studentship Projects
| Project Reference | Relationship | Related To | Start | End | Student Name |
|---|---|---|---|---|---|
| EP/S513842/1 | 01/10/2018 | 30/09/2024 | |||
| 2109081 | Studentship | EP/S513842/1 | 01/10/2018 | 31/12/2022 | Filippos Ventirozos |
| Title | Instructional Text Semantic Parsing for Ambient Intelligence |
| Description | In this study, we propose an End-User programming framework that can aid people who cook. The framework input is the text from cooking recipes which then is used to program IoT devices in the kitchen. The purpose of the current questionnaire-interview is to explore whether and to what extent this automation is helpful for cooks. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2020 |
| Provided To Others? | Yes |
| Impact | These are five series of interviews with people who cook raging from beginner to experienced challenger level. The questions of the interview aimed to uncover which processes or things during the cooking process impedes or annoys the user. And how technological innovation in the form of ambient intelligence could aid. This qualitative dataset is useful for researchers to focus the technological development for the kitchen in the right direction. |
| URL | https://figshare.com/articles/dataset/Instructional_Text_Semantic_Parsing_for_Ambient_Intelligence/1... |
| Title | Interactive Clustering of Cooking Recipe Instructions:Towards the Automatic Detection of Events Involving Kitchen Devices |
| Description | Abstract: Cooking recipes are a rich source of semantic information. They contain instructions for food preparation tasks, specifying the actions that should be carried out which typically involve various ingredients and kitchen devices. In an IoT scenario, instructions in cooking recipes can form the basis for automatically controlling kitchen devices without any programming. However, as these instructions are written in natural language, they first need to be transformed or parsed into machine-interpretable commands. As a step towards this, we investigate methods for identifying the various types of actions (events) that kitchen devices are involved in. We cast this problem as a clustering task, whereby recipe instructions involving a given device of interest, are automatically grouped according to the type of event described. Each sentence in every instruction is represented by its embedding vector which is computed using a BERT-based model, specifically one pre-trained using a Roberta architecture. We cluster these sentence embeddings using our newly proposed interactive machine learning (IML)-based framework underpinned by the HDBScan clustering technique. We demonstrate that our IML framework can detect events in sentences with satisfactory accuracy, reaching almost the same level as human performance. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2021 |
| Provided To Others? | Yes |
| Impact | This dataset was created by a couple of annotators and was used to evaluate an algorithm for clustering cooking oven and fridge commands. This work adds to the greater vision of having self programmed kitchen appliances and sensors to aid users in the kitchen. The users could benefit from being taught on how to cook better (i.e. nutrition, precision cooking) and reminding them of when their involvement is needed, particularly useful for people with dementia. |
| URL | https://figshare.com/articles/dataset/Interactive_Clustering_of_Cooking_RecipeInstructions_Towards_t... |