Breaking the Glass: Multimodal, Malleable Interactive Mobile surfaces for Hands-In Interactions

Lead Research Organisation: Swansea University
Department Name: College of Science

Abstract

Mobile phone and tablet touch screens are flat, dead surfaces. Our work seeks to explore the potential of a far more fluid, 'alive' portable display. It departs radically from existing deformable research by endeavouring to provide truly direct interaction with affordances, controls and content integrated within a visual display. We will be highly ambitious and adventurous, pushing the boundaries of technical possibility and being inspired by perspectives from deformations in the animal world through a Co-I and team from biosciences. Our work will be further grounded and informed by our end-user partner, BBC R&D. The outcome will be scenarios, interaction styles and a range of fully-functional prototypes that will enable us to map out a design space to drive developments in both display materials and a richer, expressive toolkit of gestures and manipulation on touch surfaces.

Consider the following scenarios that illustrate the possibilities of the new technology and user interfaces:

1. Sam is using the new paint application on her tablet. At the top of the screen is a row of paint pots that dynamically recess into the display. Each contains a colour, and the deeper she dips her paintbrush into the pot, the deeper and thicker she is able to paint onto the remaining screen surface, just as in the real world of watercolour sets.

2. Rosie is watching a BBC nature programme on her very large screen HD television set. In her hands she holds a tablet which can display companion content for the programme. While the large screen shows a butterfly dancing around a garden, her tablet screen deforms to provide a sensation of the flapping wings, indicating to Rosie that there is further content on the second screen. However, she is immersed with the main screen display and briskly pushes the deformation away to the right of the mobile. The force of her interaction is used by the system as a gauge to her interruptibility as additional second screen content becomes available.

3. Siân is producing a live music event for broadcast over iPlayer. She needs to keep her eyes and ears alive to the pace and spectacle of the band whilst catching the individual performances of each of its members. Normally a sea of controls from an array of mixing desks would surround her, rigged to control every potential shot from the camera crew on and off stage. The new flat panel touch screen mixers enable Siân to customise the controls, reducing the kit required, but without tactile feedback her attention is diverted from the action to the control surface. These micro moments of distraction can be critical and the continual switching of attention increases the stress of the job. The new deformable displays give her best of both worlds: a smaller, responsive mixing desk with tactile feedback that keeps her eyes on the performance she is broadcasting.

We propose a number of transformative benefits of the novel surfaces and architectures we will construct:

- The three dimensional and multimodal nature of the elements can be used to communicate features of an information space in more efficient and satisfying ways;
- The surface can provide sophisticated and nuanced controls with, for instance, the extent to which the user pokes her finger or stylus into the surface adjusting the degree of system response; and,
- The tangibility of the elements can enable eyes-free operation, allowing the user to combine their handheld interaction with other objects in their environment in order to deal with a complex content space. The platform can then provide physical affordances and controls, in a dynamic way, as the user interacts with the Internet of Things around them.

The team combines world leading researchers with expertise in: user experience, hardware and materials innovation, electronics, design space analysis and biosciences.

Planned Impact

The market for new display forms and the experiences and services they can afford is vast. Already, there are over two billion owners of smartphones, with analysts expecting this to increase dramatically over the next 5-10 years, and existing users upgrading. Our work will be the very start of a highly novel surface that could revolutionise this market in the longer term, with the successors of the materials and interaction styles we investigate finding their way onto devices everyone in the world owns. We would expect companies, including start-ups and spin-outs, to exploit this work with direct benefit to the health of the UK economy and employment. We are keen to see this project lead to such tangible outcomes, and have previously had success with transferring advanced output technologies to spin-outs (e.g., see ultrahaptics.com).

The improved ways a user can interact with ICT (from a better understanding of big data sets to eyes-free control of objects connected to the 'Internet of Things' (IoT)) will have significant impacts on the perceived quality of life of populations. For specific application areas there could be even greater benefits: for example, consider a surgeon manipulating a keyhole surgery tool during an operation where the tool is controlled by a mobile surface. As she focuses on a tumour in the patient's colon, the deformable surface generates an appropriately shaped tweezer element, the size and form adapted dynamically to the tumour's characteristics. Or, consider a partially sighted user who can read enlarged text on a mobile but has great difficulty presently in operating sliders, selectors or switches on the UI, for instance. The hands-in surface enhances the communication channel by providing a range of non-visual cues including, for example, tangible size and shape of controls.

While we expect the full and sustained impacts of our work to be in the long term (i.e., 10 years out and beyond), we expect many of our discoveries to be applied quickly via existing platforms and new devices in the short to medium terms. For example, the work that will look at providing direct manipulation of IoT-connected objects will inform implementations possible today: e.g., pointing a mobile at an overhead projector might generate a photorealistic rendering of the focusing knob on a conventional touch screen that the user can manipulate (as opposed to the tangible, deformed control possible in the advanced displays envisaged here). Alternatively, a malleable surface peripheral could be produced and connected to a conventional mobile to provide additional hands-in interactions without integration into the smartphone itself. We see, then, many varied possibilities of such products and services, and impacts on the economy within a short timeframe.

We also anticipate significant impacts on the UK's capability to lead in this emerging field of innovation. There are relatively few groups in the UK focusing on advances in materials to extend user experience and interaction (with Co-I Subramanian's being one of these). Our work will enhance the profile and standing of this important set of activities in the UK, developing people with skills and outlooks that can shape and take the opportunities that these new platforms will enable.

Publications

10 25 50
publication icon
Sahoo D (2017) JDLED

publication icon
Steer C (2017) Deformable Paint Palette

 
Description We have developed a proof of concept of a deformable screen that allows users to touch and use dials and sliders that emerge from the surface of a mobile.
Exploitation Route We are in process of creating a toolkit of examples that others can appropriate.
Sectors Creative Economy,Digital/Communication/Information Technologies (including Software),Culture, Heritage, Museums and Collections

URL http://cs.swan.ac.uk/breaking-the-glass/
 
Description The findings have been presented at the British Science Festival and also covered by BBC and Reuters. They have also been presented at the Hay Festival (2017), Cheltenham Science Festival (2018).
First Year Of Impact 2016
Sector Creative Economy
Impact Types Cultural,Societal

 
Description BBC UXRP 
Organisation British Broadcasting Corporation (BBC)
Department BBC Research & Development
Country United Kingdom 
Sector Public 
PI Contribution Worked extensively with the BBC on informing the design of attention for multi-device platforms.
Collaborator Contribution Dr. Michael Evans is my joint academic supervisor.
Impact All PhD academic papers and awards noted have been from directly working with the BBC. I have worked with my supervisor, Dr. Michael Evans, and (to a lesser extent) various content designers at the BBC to inform my research.
Start Year 2014
 
Description NESTA Digital Innovation and the Arts in Wales, invited talk: The future of the digital economy 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Policymakers/politicians
Results and Impact An invited talk about the project, discussing the project and its aims at The Digital Innovation Fund for the Arts in Wales.
Year(s) Of Engagement Activity 2016
URL http://www.nesta.org.uk/project/digital-innovation-fund-arts-wales
 
Description Project featured on Reuters Technology 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact The project was featured on Reuters Technology - "Scientists working on 'morphing' smart phone screens". This involved a video interview with the PI about the project, who outlined our vision.
Year(s) Of Engagement Activity 2016
URL http://reut.rs/2dmihV2
 
Description Touching the Future talk at British Science Festival 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Public/other audiences
Results and Impact Prof. Matt Jones, accompanied by Simon Robinson, Timothy Neate, Cameron Steer and Jen Pearson gave a talk at the British Science Festival (https://www.britishsciencefestival.org/)

The talk was given to a lecture hall of the general public at Swansea University and talked about our past work and our vision for this project.

The event was covered on the BBC's Science Cafe, BBC Click, and by the South Wales Evening Post.
Year(s) Of Engagement Activity 2016
URL http://www.bbc.co.uk/programmes/b07tmbk2#playt=1m11s