Design the Future 2: CrowdDesignVR

Our initial proposal CrowdDesign set out to explore how we can aid rapid prototyping of mobile sensor-based user interfaces by exploiting the versatile sensor capabilities of mobile phones. The primary objective was to investigate if we can crowdsource such sensor-dependent tasks to mobile devices in order to assist designers in rapidly evaluating new interaction techniques in situ. We have identified a very strong research trajectory that motivates continuing the CrowdDesign project beyond this year: CrowdDesignVR. In this follow-up project we propose to substantially extend the scope of the CrowdDesign project and elevate it from the smartphone platform and into a virtual reality platform. To enable the exploration of a very promising research trajectory for crowdsourced human-computer interaction, we need to invest time and effort into realising a high-quality crowdsourcing platform for VR.

CrowdDesignVR will be the first crowdsourcing system for virtual reality. It will distribute tasks across the Steam VR distribution network, which allows it to reach a large sample of VR users. Prior research cannot reach this scale, as research has been limited to opportunity-sampling local participants and then train them to use a specific VR system. In contrast, by enabling access to thousands or even tens of thousands of Steam users, CrownDesignVR facilitates user interaction data collection at a scale that is several orders of magnitudes larger. This provides a number of wider benefits: 1) we can during the course of the project create more accurate models of human actions; 2) we can collect sufficient training data to train machine learning models, such as deep neural network models to accurately decode common user interface interaction patterns, such as typing, gesturing and determining whether an action was intended or not by the user.

Since crowdsourcing tasks in a high-fidelity VR environment is a new avenue of research, there are many fundamental questions that need to be answered. We believe this project could result in potential seminal work on the understanding of the design space for crowdsourcing in VR.

Another potential impact is the data itself. Our internal work on building deep neural networks for decoding typing tasks on touchscreen and physical keyboards has revealed that deep neural networks (specifically, recurrent neural networks) output traditional hidden Markov model decoding. However, we have also found that the amount of data that needs to be collected is very large, in fact, we use our CrowdDesign task architecture as mentioned previously in our report to collect touchscreen data from hundreds of users. CrowdDesignVR can substantially widen the scope and let us tackle some of deep previously unsolved questions in user interface design, such as how we can build a gesture recogniser that is capable of learning to recognise both open-loop (direct recall from motor memory) and closed-loop (visually-guided motion) gestures on both the 2D plane and in 3D space. A large amount of data would allow us to train a recurrent neural network to learn this separation. The potential is large as users are always in a continuum between open-loop and closed-loop interaction. However, due to the fundamental differences in the underlying generative models that result in the observed behaviour, it is very difficult to collect sufficient training data in lab.

Academic Dissemination
We will strive to publish in the top venues in our discipline, the CHI conference and ACM Transactions on Computer-Human Interaction (TOCHI) primarily. Other venues we will consider are the Ubicomp, UIST and ISMAR conferences and IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI). We will also propose a workshop or tutorial at CHI to invite other researchers to explore this research area.

Ensuring Lasting Impact of the Project
At month 30 we will assess the success of the platform with the assistance of our industry partners. If the platform has a substantial uptake we will investigate commercial or non-profit avenues for enabling the platform to stay open. Our experience is that it is very difficult to maintain a platform without recurring income funding it. Therefore, if the platform is to last we are going to develop a business model for sustaining it in consultation with Cambridge Enterprise, who has advised on similar models before, such as Granta Design. We may also choose to explore a partnership with our industry partners.

We also believe there is a potential to excite, entice and engage the players on the platform themselves who are using the platform. We are all affected by user interface issues and many users are interested in the mechanisms behind them. Therefore, in line with the ethos of the CrowdDesign project, we will attempt to treat users on the platform as first-class citizens when it is opened up so that they themselves can explore user interface tasks on the platform they are using.

Finally, we will strive to release data openly using the Cambridge open data repository.

University and Departmental Support
The University of Cambridge and the Department of Engineering has experience in supporting technology start-ups and other translational activities. We will consult with Cambridge Enterprise to explore avenues for potential commercial or non-profit exploitation of the data, models, algorithms and platform.