Breaking the Glass: Multimodal, Malleable Interactive Mobile surfaces for Hands-In Interactions

Mobile phone and tablet touch screens are flat, dead surfaces. Our work seeks to explore the potential of a far more fluid, 'alive' portable display. It departs radically from existing deformable research by endeavouring to provide truly direct interaction with affordances, controls and content integrated within a visual display. We will be highly ambitious and adventurous, pushing the boundaries of technical possibility and being inspired by perspectives from deformations in the animal world through a Co-I and team from biosciences. Our work will be further grounded and informed by our end-user partner, BBC R&D. The outcome will be scenarios, interaction styles and a range of fully-functional prototypes that will enable us to map out a design space to drive developments in both display materials and a richer, expressive toolkit of gestures and manipulation on touch surfaces.

Consider the following scenarios that illustrate the possibilities of the new technology and user interfaces:

1. Sam is using the new paint application on her tablet. At the top of the screen is a row of paint pots that dynamically recess into the display. Each contains a colour, and the deeper she dips her paintbrush into the pot, the deeper and thicker she is able to paint onto the remaining screen surface, just as in the real world of watercolour sets.

2. Rosie is watching a BBC nature programme on her very large screen HD television set. In her hands she holds a tablet which can display companion content for the programme. While the large screen shows a butterfly dancing around a garden, her tablet screen deforms to provide a sensation of the flapping wings, indicating to Rosie that there is further content on the second screen. However, she is immersed with the main screen display and briskly pushes the deformation away to the right of the mobile. The force of her interaction is used by the system as a gauge to her interruptibility as additional second screen content becomes available.

3. Siân is producing a live music event for broadcast over iPlayer. She needs to keep her eyes and ears alive to the pace and spectacle of the band whilst catching the individual performances of each of its members. Normally a sea of controls from an array of mixing desks would surround her, rigged to control every potential shot from the camera crew on and off stage. The new flat panel touch screen mixers enable Siân to customise the controls, reducing the kit required, but without tactile feedback her attention is diverted from the action to the control surface. These micro moments of distraction can be critical and the continual switching of attention increases the stress of the job. The new deformable displays give her best of both worlds: a smaller, responsive mixing desk with tactile feedback that keeps her eyes on the performance she is broadcasting.

We propose a number of transformative benefits of the novel surfaces and architectures we will construct:

- The three dimensional and multimodal nature of the elements can be used to communicate features of an information space in more efficient and satisfying ways;
- The surface can provide sophisticated and nuanced controls with, for instance, the extent to which the user pokes her finger or stylus into the surface adjusting the degree of system response; and,
- The tangibility of the elements can enable eyes-free operation, allowing the user to combine their handheld interaction with other objects in their environment in order to deal with a complex content space. The platform can then provide physical affordances and controls, in a dynamic way, as the user interacts with the Internet of Things around them.

The team combines world leading researchers with expertise in: user experience, hardware and materials innovation, electronics, design space analysis and biosciences.

The market for new display forms and the experiences and services they can afford is vast. Already, there are over two billion owners of smartphones, with analysts expecting this to increase dramatically over the next 5-10 years, and existing users upgrading. Our work will be the very start of a highly novel surface that could revolutionise this market in the longer term, with the successors of the materials and interaction styles we investigate finding their way onto devices everyone in the world owns. We would expect companies, including start-ups and spin-outs, to exploit this work with direct benefit to the health of the UK economy and employment. We are keen to see this project lead to such tangible outcomes, and have previously had success with transferring advanced output technologies to spin-outs (e.g., see ultrahaptics.com).

The improved ways a user can interact with ICT (from a better understanding of big data sets to eyes-free control of objects connected to the 'Internet of Things' (IoT)) will have significant impacts on the perceived quality of life of populations. For specific application areas there could be even greater benefits: for example, consider a surgeon manipulating a keyhole surgery tool during an operation where the tool is controlled by a mobile surface. As she focuses on a tumour in the patient's colon, the deformable surface generates an appropriately shaped tweezer element, the size and form adapted dynamically to the tumour's characteristics. Or, consider a partially sighted user who can read enlarged text on a mobile but has great difficulty presently in operating sliders, selectors or switches on the UI, for instance. The hands-in surface enhances the communication channel by providing a range of non-visual cues including, for example, tangible size and shape of controls.

While we expect the full and sustained impacts of our work to be in the long term (i.e., 10 years out and beyond), we expect many of our discoveries to be applied quickly via existing platforms and new devices in the short to medium terms. For example, the work that will look at providing direct manipulation of IoT-connected objects will inform implementations possible today: e.g., pointing a mobile at an overhead projector might generate a photorealistic rendering of the focusing knob on a conventional touch screen that the user can manipulate (as opposed to the tangible, deformed control possible in the advanced displays envisaged here). Alternatively, a malleable surface peripheral could be produced and connected to a conventional mobile to provide additional hands-in interactions without integration into the smartphone itself. We see, then, many varied possibilities of such products and services, and impacts on the economy within a short timeframe.

We also anticipate significant impacts on the UK's capability to lead in this emerging field of innovation. There are relatively few groups in the UK focusing on advances in materials to extend user experience and interaction (with Co-I Subramanian's being one of these). Our work will enhance the profile and standing of this important set of activities in the UK, developing people with skills and outlooks that can shape and take the opportunities that these new platforms will enable.