Intelligent Mobile Crowd Design Platform

Lead Research Organisation: University of Cambridge
Department Name: Engineering

Abstract

We propose CrowdDesign: a novel crowdsourcing-based design/evaluation platform that allows evolving design concepts with heterogeneous user groups, provides context-aware analytics and leverages end-users' creativity. A central notion in CrowdDesign is a designlet-a specific user interface design, such as a prototype of a heating control interface or a smart remote control, coupled with a simple user task (such as setting the desired temperature or choosing a channel). To ease the development of designlets, CrowdDesign will be built on top of Google Android's existing development frameworks and extend their high-level and flexible design environment in order to minimise coding efforts. A designer submits a designlet via the CrowdDesign platform, which then distributes it to thousands of volunteers who are using the CrowdDesign mobile app. This app then weaves designlets into an engaging workflow in the app, which will be designed to entice, engage and retain volunteers using a variety of social features and gamification elements. Usage data from participants tackling the prescribed task in the designlet is then provided to the designer via a web-based analytics platform. Contextual data gathered via on-board mobile device sensors will be provided to the designer in order to understand broad interaction contexts, such as whether a participant was standing still or walking when engaging with their design. CrowdDesign relies on intrinsic motivation to entice, engage and retain users.

The fundamental problem this proposal is tackling is the difficulty of evaluating user interface designs in realistic use contexts with large representative groups of users. This is critical in order for designers to fully emphasise with end-users. Unfortunately it is also time-consuming and expensive to recruit participants for lab studies and such studies often fail to capture important design implications. CrowdDesign makes it easier for designers to perform rapid in-situ evaluations of new user interface designs based on primarily three observations: 1) a large subset of current user interfaces can be accurately modelled by commodity smartphones and tablets. For instance, such user interfaces are commonly used in a) mobile apps, b) interfaces for users with motor disabilities and c) home appliance interfaces, such as heating controls; 2) mature sensor frameworks enable us to automatically sense the context of use, which further helps designers emphasise with end-users; and 3) prior research has convincingly demonstrated it is possible to crowd source tasks to tens of thousand of users by relying on users' intrinsic motivations. However, prior research has not established the feasibility of interleaving a crowdsourcing platform with a user interface design process and has not attempted to make participants of the platform first-class citizens.

Planned Impact

Near the end of the project (starting in month 15) we will explore routes to ensure the CrowdDesign platform lives on beyond this project. This process will be aided by departmental support and a scheduled roadmapping session in month 15 (see below). We will then assess the following, and other, routes toward lasting impact:
Open source platform: If we choose this alternative we will set up a sustainable community of researchers and volunteers to maintain, improve and monitor the platform. The PI has some previous experience in this direction, although not at the scale required for CrowdDesign. If this route is chosen the PI will seek advice from Prof. David MacKay at the PI's department as Prof. MacKay has extensive experience in releasing and maintaining research software as open source.
Technology start-up: If we choose this alternative we will attempt to launch a company to monetise the platform. The PI has previously co-founded a technology start-up called ShapeWriter, Inc., which has been successfully acquired by the market leader in its space (Nuance Communication, Inc.). The PI has experience in raising money for technology start-ups. If this route is chosen we will also rely on support Cambridge Enterprise and advice from the Director of Engineering and the Knowledge Transfer Facilitators (KTFs) at the Department of Engineering.

The University of Cambridge has an established network of fully-funded KTFs, including three within Engineering. Each with decades of experience in industry, business and finance, these KTFs are experienced in creating impact and ensuring good knowledge transfer, working with academic teams to support them in realising the impact potential of their research. The Department also has a Director of Research, whose experience and knowledge adds to the KTFs' expertise. They will work closely, throughout the 18 months of the project, both with the investigators and also with the industrial partners to enable a wide range of impact activities take place. By providing industrial and academic liaison, market research and consultancy skills to the study, they will increase effectiveness of the Initiative's impact agenda and raise the levels of knowledge transfer achieved. The Department of Engineering has a strong and successful history of outreach. The Department has a dedicated Outreach Officer, who co-ordinates, organises and manages activities, ensuring a wide and diverse engagement programme.
Cambridge Enterprise (CE), a wholly owned subsidiary of the University of Cambridge, is expert in technology transfer, consultancy and seed fund capital. CE will advise on routes, options and methods of engagement and commercialisation. CE and the KTFs will support the project in all aspects of commercialisation and industrial engagement. The project will include a professionally-run technology roadmapping session scheduled toward the end of the project. The workshop will bring together academics and industry to identify new avenues for commercialisation, focus future research and prioritise the next steps for the project. The IfM ECS (a wholly owned subsidiary of the University of Cambridge) will design and run the workshop. It is a world centre of excellence for roadmapping. It has been researching the application of roadmapping for more than a decade and its techniques have helped more than 250 public and private sector organisations with their strategic and technology innovation planning.
 
Description We discovered it is possible to efficiently crowdsource data collection from sensor-based mobile user interfaces. We used the microtask market Amazon Mechanical Turk and created a framework that enabled web-friendly microtasks. Using our framework we surveyed crowd workers on Amazon Mechanical Turk and we discovered that they were generally willing to carry out work on microtasks using their mobile devices and would even switch from a laptop or desktop computer to a mobile device when prompted. We probed their mobile device sensors and found we could read the most pertinent sensors using our framework. We then crowdsourced two microtasks which rely on ms; 4) The most pertinent mobile device sensors can be read using web APIs provided by the largest microtask market in the world---Amazon Mechanical Turk.obile device sensors (touchscreen gesture data and side-taps on the phone detected by reading linear accelerometer data) and found we could rapidly collect data from over a hundred people. The key findings for this part of our project are thus: 1) Crowd workers are happy to carry out microtasks on their mobile devices; 2) A subset of crowd workers are already doing crowd work using their mobile devices---they enable designers and researchers to carry out in-situ contextual research; 3) The vast majority of crowd workers are situated in front of a laptop or desktop computer and switch to their mobile device when prompted---this pool cannot be relied upon when carrying out in-situ research for mobile device. We then discovered it is possible to crowdsource data and simultaneously control the interaction context using Virtual Reality (VR). We developed a game with a compelling narrative to entice and engage participants and by participating in the game participants are carrying out controlled interaction tasks which can be used as design knowledge in a design process. This finding and the subsequent demonstration of the game lead to the successful follow-up funding from EPSRC (Design the Future 2), which is the currently ongoing project. As part of this VR exploration we have made several discoveries which have been accepted for publication but are not yet attributable in Researchfish. The findings include it is possible to allow users to type on a physical keyboard in VR and on a touchscreen keyboard in VR and retain a large proportion of their typing ability. In addition, we have investigated fundamental human performance as defined by human action laws and discovered that crossing actions form a suitable basis for typical button presses in VR and in Augmented Reality (AR) interfaces.

In addition, as part of this project we did a longitudinal survey of the employment characteristics of crowd workers on a micro-task mark, the first of its kind in terms of its scale and methodology which has been accepted as a full paper at CHI 2019 (DOI not yet active so not reported yet as a publication).
Exploitation Route The findings allow interface designs to rapidly prototype interaction tasks with a larger and more appropriate sample of users who are using the technologies in-situ.
Sectors Digital/Communication/Information Technologies (including Software)

 
Description The major impact of this 18-month proposal was to seed a longer 36-month proposal 'Design the Future 2: CrowdDesignVR' (EP/R004471/1), which have its own impacts. Apart from seeding a larger proposal, this specific proposal had three non-academic impacts: 1. The paper 'Observations on Typing from 136 Million Keystrokes' demonstrated the efffiacy of mobile crowdsourcing to gather a large amount of behavioural typing data, which lead to further work (in the follow-on project) that culminated in a subsequent publication that was able to use mobile crowdsourcing to create large-scale study of mobile typing behaviuour. Among other things that study found that the fastest way to type was to use both thumbs, turn on auto-correct and turn off word predictions (the latter 'negative' feature was subsequently studied by my group in a paper at CHI 2021, which was also featured in The Conversation). The follow-up work lead to an interview by The New York Times, reported as part of the follow-up proposal (EP/R004471/1). 2. The paper 'A review of user interface design for interactive machine learning' has attracted a large number of citations (151 citations since 2018) and become the "go to" paper for understanding current issues in interactive machine learning. It was also highlighted in a blog post ('Pay Attention to the Man Behind the Curtain': https://medium.com/acm-tiis/pay-attention-to-the-man-behind-the-curtain-dc4971419239). 3. The paper 'The office of the future: virtual, portable and global' lead some early foundational work to the idea of moving office work into Virtual Reality. This lead to the several follow-on publications in the subsequent grant propsoal and to a collaboration with Microsoft Research which is still active.
First Year Of Impact 2019
Sector Creative Economy,Digital/Communication/Information Technologies (including Software)
Impact Types Cultural,Societal

 
Description Design the Future 2
Amount £560,504 (GBP)
Funding ID EP/R004471/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 01/2018 
End 12/2020
 
Description Shanghai Trade Expo 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Industry/Business
Results and Impact Invited talk at large trade expo in Shanghai.
Year(s) Of Engagement Activity 2017