Swarm Social Sound Modulators

Lead Research Organisation: University College London
Department Name: Computer Science


The UK is a world-leader in creating interactive applications that are enabled by computational manipulation of acoustic wave fronts. Three examples of such applications include mid-air haptics, directional audio, and volumetric 3D particle displays. Using a phased array of ultrasonic speakers that are precisely and individually controlled, we can create high pressure focal points in mid-air. Modulating these focal points in various ways, it is possible to 1) create rich tactile sensations when touched by the bare hands, 2) steer directional sounds that propagate over long distances un-attenuated, 3) levitate small particles that when rapidly moved in space emulate volumetric 3D shapes due to a phenomenon called persistence of vision.

The exploitation of these amazing new technologies is uniquely available to Ultraleap (a UK based company) and Sussex University who have a long and productive history of collaboration. For example, Ultraleap is currently combining hand-tracking and ultrasonic mid-air haptic feedback solutions for applications ranging from VR training simulators, automotive interfaces, gaming machines, and next generation digital signage kiosks. Similarly, Sussex University is creating multimodal 3D displays based on rapidly updating ultrasonic phased arrays to create persistence of vision when moving acoustically levitated objects. A significant constraint to the wide-scale deployment of the underlying technology of phased arrays is the cost and complexity of a non-modular system because it limits applicability. For example, there is no one size fits all phased array with most integrated solutions needing to be custom developed.
In this project, we will circumvent such problems altogether by creating simple and low cost modular spatial sound modulator (SSM) units i.e. smaller arrays of acoustic sources, to be placed around the interactive space, that can collectively out-perform the single large monolithic solution we currently have. Moreover, we will take a leap forward in sound-field control by removing scalability and reusability issues thus opening up the exploitation of phased array technologies into other applications domains that can benefit from the non-contact delivery of haptic feedback, steerable directional sound, and/or volumetric 3D particle displays. Specifically, we will draw on the well-developed literature of multi-agent game theory and distributed computing and use them to build a decentralised swarm architecture that can flexibly accommodate numerous SSM units. Each SSM will emit and modulate the sound-field nearby it while sharing a common awareness of the contextual details with its swarm host, while the desired collective behaviour will emerge from the interactions between multiple SSMs and their interactions with the environment.

There are several anticipated benefits to our proposed approach. Firstly, by designing simple, independent SSMs we are able to address multiple commercial applications using the same primitive unit, while simultaneously streamlining the manufacturing pipeline. These modular units can be used individually or combined in a myriad of ways to create new applications. Secondly, by enabling a distributed control architecture, swarm SSMs can seamlessly and progressively scale up. By incrementally combining larger numbers of modular units, customers can initially use a small number of SSM units, and dynamically grow the capabilities of their interactive multimodal system by adding new devices according to the application needs. Finally, by using game theory we will enable SSMs to dynamically cooperate with each other as to meet application objectives independent of the application logic and the arrangement of our modular devices thus simplifying the development and design process and enabling creative designers to focus on the delivery of evermore immersive and multimodal experiences.


10 25 50
publication icon
Montano-Murillo R (2023) OpenMPD: A Low-Level Presentation Engine for Multimodal Particle-Based Displays in ACM Transactions on Graphics

Description CES 2023 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Industry/Business
Results and Impact Ultraleap demonstrated teh distributed SSM layout at CES 2023. The distributed layout allows for a more flexible transducer arrangement within a car interior while also minimizing its footprint. Moreover, the demo application developed underwent significant UX considerations and user-testing while following industry trends for user interfaces (UIs) akin to Apple's CarPlay and Google's Android Auto and the use of secondary heads-up displays (HUDs).
Year(s) Of Engagement Activity 2022,2023
Description Interview for BBC Earth 
Form Of Engagement Activity A broadcast e.g. TV/radio/film/podcast (other than news/press)
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Schools
Results and Impact BBC Earth filmed in our lab to discuss can sound kill. As part of that program we showed them about levitation and described the magical world of sound.
Year(s) Of Engagement Activity 2022
URL https://youtu.be/Deu_FD2TGlA
Description Keynote at VRST 2021 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Industry/Business
Results and Impact Diego Martinez Plasencia gave a keynote talk at a leading virtual reality conference in Japan. The talk titled "Walk-up-and-use experiential interactions in Mixed-Reality". The talk describes how we built ultrasonic speaker arrays to suspend tiny objects in the acoustic field and manipulate them to create persistence of vision displays. We do this by computing acoustic phase-only holograms that are delivered using the speakers to shape the wavefront. Similarly, we create tactile sensations by focusing the waves on the palm of the user. All these examples create sensation in mid-air - so users do not have to touch or hold any device to experience it.
Year(s) Of Engagement Activity 2021
URL https://programs.sigchi.org/vrst/2021/program/content/67310