User Interaction with self-supporting free-form physical objects

The primary goal of this project is enhance human-computer interaction by dynamically creating and manipulating physical shapes by levitating and moving a large collections of lightweight 3D objects using principles of acoustic levitation. The proposed idea is illustrated in Figure 1 where one ultrasound transducers is placed as a floor mat and a large collection of polystyrene beads create the shape of a dog that can wag its tail.

This will enable us to represent complex datasets in a physical form and allow users to dynamically manipulate it. We are now moving away from traditional human-computer interaction techniques like buttons, keyboards and mice to touch (e.g., Smartphones and multi-touch gestures) and touchless interactions (as with the Kinect and Leap Motion controllers). The limiting factor in these new forms of interactions is, ironically, the lack of any physicality and lack of any co-located feedback. One has no controller or interface element to physically touch and interact with and the visual feedback that may be available is disconnected from the location of the gesture.

In our vision, the computer will control the existence, form, and appearance of complex levitating objects composed of "levitating atoms". Users can reach into the levitating matter, feel it, manipulate it, and hear how they deform it with all feedback originating from the levitating object's position in mid-air, as it would with objects in real life. This will change how people use technology as they can interact with technology in the same way they would with real objects in their natural environment.

We see many possible benefits of physicalisations for the individual and society: they make data more accessible by leveraging our perceptual exploration skills via active perception, depth perception, non-visual senses, and intermodal perception; give novel exploration possibilities to the visually impaired; support learning via cognitive benefits of direct manipulation of physical artefacts; bring data to the real world for communication and exhibition and; and finally act as tools for engaging audiences with information.

One of the aims of this project is to explore the roles levitation and physicalisations can play in public engagement with science, including public engagement with areas of policy that have technical content such as energy and climate change. This project has practical roles to play in supporting and fostering dialogue as well as in provoking interest in technical topics. The key will be to integrate the principles learned from WP1 - WP3 to engage the public with good storytelling.

Through the collaboration of Carbon Visuals, an Installation Artist will be responsible for creating novel interactive art installations that exploit the various research tools and systems developed by the project. Our system will empower new approaches to explore, manipulate and understand complex 3D representations of different types of data and interactive objects. By providing direct manipulation of physical objects in a 3D space combined with rich tactile feedback, users will be able to feel rather than simply look at their interactions. Consequently, researchers and interaction designers will be able to take advantage of the increased engagement to offer new powerful active exploration techniques with physical representations that have the potential to stimulate reasoning and analytical skills in new ways. This new interaction paradigm will also diminish the transition cost between novice and expert, allowing users to reach high performance for a small learning investment. Since direct 3D manipulation relies on our natural human skills for interacting with physical objects, the new interaction techniques will be easier to learn and master than their desktop computer counterparts using mice or keyboards. This will extend users' abilities to resolve complex 3D spatial-based problems quickly.

Our vision enables one to represent complex datasets in a physical form allowing users to analyse and perceive complex time-varying 3D structures through physicalisations. This idea to represent data in physical form is not new. Physicalisations as stone or pebble tokens have already been used thousands of years ago. More than 7000 years ago, the Sumerians, for example, used clay tokens to represent quantitative data. Even today, physicalisations in the form of data sculptures are created and used by designers and artists as alternate methods for conveying datasets. They have projected communicative impact, quickly and easily helping users to understand, for example, how crime rates vary across a city or how an urban development will change the environment. Similarly, scientists use and have used physicalisations to help analyse and perceive complex time-varying 3D structures. For example, James Maxwell famously created physicalisations of thermodynamic surfaces to help perceive isopiestics and isothermals. This helped Maxwell visualize Gibbs' thermodynamic surface, which expressed the relationship between the volume, entropy, and energy of a substance at different temperatures and pressures.

However physicalisations in the past have lacked dynamicity, automation, and computation all of which can be enabled by this project. We see many possible benefits of physicalisations for the individual and society: they make data more accessible by leveraging our perceptual exploration skills via active perception, depth perception, non-visual senses, and intermodal perception; give novel exploration possibilities to the visually impaired; support learning via cognitive benefits of direct manipulation of physical artefacts; bring data to the real world for communication and exhibition and; and finally act as tools for engaging audiences with information.