Supporting Feature Engineering for End-User Design of Gestural Interactions

Sensors for analysing human gesture and activity (such as accelerometers and gyroscopes) are becoming increasingly affordable and easy to connect to existing software and hardware. There is great, unexplored potential for these sensors to support custom gestural control and activity recognition systems. Applications include include the creation of bespoke gestural control interfaces for disabled people, new digital musical instruments, personalised performance analysis systems for athletes, and new embodied interactions for gaming and interactive art. The ability to easily create novel interactions with motion sensors also benefits schoolchildren and university students who are learning about computing through the use of sensors with platforms such as BBC micro:bit and Arduino.

We have previously established methods for enabling people without programming expertise to build custom gesturally-controlled systems, using interactive machine learning. These methods allow people to easily create new systems by demonstrating examples of human actions, along with the desired label or computer response for each action.

Unfortunately, many compelling applications of custom gesture and activity recognition require substantial pre-processing of raw sensor data (i.e., "feature engineering") before machine learning can be applied successfully. Experts first apply a variety of signal processing techniques to sensor data in order to make machine learning feasible. Many people who would benefit from the ability to create custom gestural interactions lack the signal processing and programming expertise to apply those methods effectively or efficiently. It is not known how to successfully expose control over feature engineering to non-experts, nor what the trade-offs among different strategies for exposing control might be.

Our hypothesis is that it is possible to support creation of more complex gestural control and analysis systems by non-experts. We propose to develop and compare three methods for exposing control over feature engineering to non-experts, each requiring varying degrees of user involvement.

The first method uses a fully automated approach, which attempts to computationally identify good features. The second method first elicits high-level information about the problem from the user, and then employs this information to better inform the automated approach. The third method directly involves the user in the feature engineering process. By leveraging users' ability to demonstrate new gestures, identify patterns in visualisations, and reason about the problem domain - as well as computers' ability to employ users' demonstrations to propose new relevant features - this interactive approach may yield more accurate recognisers. Such an approach may also help users learn about the utility of different features, enabling more efficient debugging of their systems and a better understanding of how to build other systems in the future.

We are interested in understanding both the accuracy and usability of these methods. We will evaluate each method with people training several types of gesture and activity recognisers. We will compare each method in terms of the accuracy of the final system, the time required for users to build the system, users' subjective experiences of the design process and quality of the final system, and improvements in users' ability to reason about building gestural interactions with sensors.

This research will enable a wider range of people to successfully use popular motion sensors to create bespoke gestural control and analysis systems, for use in a broader range of applications. Our techniques will be directly implemented in existing open-source software for interactive machine learning. The methods and study outcomes will also inform future work to support feature engineering for users creating real-time systems with other types of sensors.

This project puts at its heart user-centred design and stakeholder engagement, in order to ensure that benefits are recognised in a number of different target audiences.

1. User groups in the general public:

New groups of people will gain the ability to create compelling, customised interactions with motion sensors, including:
- People with disabilities who can create new, customised gestural interfaces for controlling computers and games;
- Musicians creating new digital musical instruments;
- Athletes and trainers creating personalised performance monitoring and analysis systems;
- Artists creating new interactive experiences;
- "Quantified self" practitioners seeking new ways to understand or change their habits;
- "Hackers" and "makers" developing new interactions with sensing hardware for practical or creative use.

The ability to create more sophisticated gesture and activity recognition systems can enhance such people's quality of life, providing an increased sense of agency, new creative outlets, and a better understanding of themselves. We will integrate our techniques for end-user feature engineering directly into existing free software that provides GUIs for use by non-programmers. This means the above users will immediately be able to employ these techniques.

2. Users in the creative and digital industries

Individuals who currently design interactions as part of their professional practice-including many artists, musicians, and indie game designers-will be able to use sensors more effectively and efficiently in their work. Other people who currently employ sensors for non-commercial, personal use, such as hackers/makers, may become more likely to commercialise their designs as the sophistication of interactions they can create improves.

3. School students using sensors to learn about computing.

The introduction of 1 million BBC micro:bits into UK classrooms this year reflects recognition that work with sensors can engage students in learning computing, encouraging creative exploration of projects that connect computing to the physical world. Our project greatly expands the set of interactions that non-experts can easily build using micro:bit's accelerometers, and our findings will directly inform the design of future tools for students experimenting and creating interactions with sensors. The software tools released by our project will also be immediately usable by university students in physical computing classrooms who are building new interactions with sensors and platforms such as Arduino and Raspberry Pi. Increased student engagement with electronics presents opportunities for additional, longer-term economic benefits: students who build solid foundations in STEM skills will be better prepared to contribute to the 21st century workforce and to become technology innovators themselves.