Content creation for Mixed-Reality Particle Based Display Exhibitions

Science fiction movies, such as Star Wars or The Avengers portray the vision of volumetric systems in which the characters cannot only see 3D content floating in mid-air, but they can also feel it and hear it, interacting with it with their bare eyes, ears and hands, just as they would do with any other real object around them.

Current display approaches, such as VR, allow us to get a glimpse of such experiences, but users need to wear headsets and other devices such as VR controllers or data-gloves. These devices create visual, tactile and auditive stimuli as to trick our brains into believing that the contents we are seeing are real.

Instead of creating illusions, Particle Based Displays (PBDs) take an approach which is much closer to these visions from science fiction, directly controlling matter in the real world to create contents that we can directly see, hear and touch. PBD systems make use of ultrasounds to trap and move small particles in mid-air. By quickly moving and colouring these particles in 3D space, PBD systems can create coloured volumetric shapes visible by our naked eyes. Making use of the pressure delivered by the ultrasound acoustic waves, PBDs can also create points of high pressure that our bare hands can feel, or induce air vibrations that create audible sound.

During this 3 year programme, our team of artists and researchers will make use of PBDs to create public artistic installations in UK and Shanghai, showcasing immersive mixed-reality installations in which virtual content (created by the PBD) and real contents (scenery, real props, illumination) will merge seamlessly, creating new and magical experiences for entertainment, story-telling, retail and digital signage.

The programme will also exploit collaborations with media artists and industries in UK, Shanghai and across the globe, creating a network of collaborators to explore the potential of PBD displays, either together with us or independently. At the same time, we will work in creating development tools to make the creation of PBD experiences easier, allowing our technology to become accessible to more people, and as a necessary step to take our PBD displays from our labs and exhibitions and into our workplaces and homes.

With the ability to control matter to create contents that users can simultaneously see, hear and touch, PBDs certainly provide a huge potential. This proposal provides a unique opportunity to explore such potential, stimulating a whole new way of thinking about 3D displays and providing the public exposure required to drive demand for such an innovative technology. This will facilitate outcomes in three main areas, beyond the academic domain:

-Economic and industrial impact: Our project can have a major impact on industry, by demonstrating potential of PBDs in large interactive installations, increasing the TRL of our base technology and providing the content creation tools to make exploitation of PBDs accessible. This can be picked up by major technology companies, either based in UK or with UK branches (SONY, Microsoft, HP labs, Ultraleap), which are showing increasing interest in the field of 3D systems. By building on our exhibitions and demonstrations to tailor their application towards successful domains, our research could lead to high impact in the national and international economy. Partnering our technology with Ultraleap (ai.e.world leading company in the creation of mid-air tactile sensations) ensures that UK will have a leading position in the exploitation of these systems. Our IP protection and dissemination strategies (i.e. showcasing at venues such as CES or FoST, with strong impacts in core markets) will provide us with a privileged position to explore and exploit commercial possibilities.

-Society: Our technology and explorations will facilitate integration of mixed-reality content (visual, auditive, tactile) into everyday tasks, without requiring additional eye-wear or devices on the users. Beyond applications for public exhibitions, theme parks or entertainment (driving domains for technologies such as VR), our technology can empower 3D interaction and exploration in other domains. The possibility to engage with traditional learning materials (e.g. paper notes, computers, miniature representations), while actively engaging with 3D representations(i.e. manipulating in 3D, directly visualizing/hearing/feeling consequences) has been proved to have positive effects in learning, with the potential to stimulate students' reasoning and analytical skills in new ways. Applications in professional domains would allow groups of experts directly experience and interact in 3D while being able to quickly break apart and engage in other necessary tasks (e.g. take notes, operate instruments, talk to peers). By removing technological boundaries to mixed-reality content, this could stimulate exploratory attitudes and creative processes that can lead to important advances (e.g. isolation of graphene). Our public installations inviting media artists and relevant industries will help us gauge potential application contexts for PBDs, inspire the general public and spark their imaginations.

-People: Output of trained people will also be an important outcome of the project. The RA, PhD, technician, but also collaborating media artists in Shanghai and UK will develop skills for the creation of PBD systems and create an initial network to exploit this space. Our permanent exhibitions will act hubs to allow media artists to further explore and showcase new experiences, as well as to maintain and grow this network of professionals. This will help in providing a pipeline of trained professionals for PBD systems, but can also help address the shortage of trained people in related fields such as VR/AR (i.e. a current demand from the market), or other types of 3D interactive systems (e.g. Ultraleap, Microsoft, Sony). Such expertise underpins the value-added manufacturing and knowledge economy that directly benefits the UK, and provides an attractive local medium for innovative companies.