Physics-Informed Neural Sound Synthesis

Digital musical instruments are ubiquitous in the music world today, but the key challenge of accurately replicating the timbres of acoustic instruments still remains. Two distinct approaches to the emulation of acoustically-produced sound in the digital domain have emerged - physics-based modelling and machine learning.

In physics-based modelling, sound is generated by a computer simulation of a physical model of a musical instrument, requiring expert knowledge about the internal dynamics of a musical system. Computer simulation methods suffer from inherent inaccuracies and high computational load, making real-time implementation impractical for all but the simplest systems. In contrast, machine learning presents a flexible and general approach to reproduce the sound of musical instruments by "learning" from their recordings. However, machine learning methods often lack interpretability and the easy user control of a synthesis model, making their performance capabilities very limited.

In the proposed project, the objective is to combine traditional physics-based simulation methods and modern machine learning approaches to achieve the ultimate musical usability of digital musical instruments. The synthesis models to be developed here will build on the complementary strengths of both approaches in order to achieve highly accurate simulations of acoustic musical instruments. These simulations will be realisable in real-time using commercial-grade hardware and would allow for meaningful user control, just as in the performance of acoustic musical instruments. Beyond this goal, it will be possible to open the door to synthetic sound from new imagined instruments without a real-world counterpart but yet retaining a natural acoustic quality.

As a result, musicians will have access to state-of-the-art simulations of acoustic musical instruments, that could be experienced on their already owned computer, and have a scope to design their own novel digital instruments and to experiment with sounds that these instruments can produce. By developing freely downloadable digital instruments, this project will bridge the gap between amateur and professional musicians and provide the means to make music to anyone and anywhere.