Making Sense of Sounds

In this project we will investigate how to make sense from sound data, focussing on how to convert these recordings into understandable and actionable information: specifically how to allow people to search, browse and interact with sounds.

Increasing quantities of sound data are now being gathered in archives such as sound and audiovisual archives, through sound sensors such as city soundscape monitoring and as soundtracks on user-generated content. For example, the British Library (BL) Sound Archive has over a million discs and thousands of tapes; the BBC has some 1 million hours of digitized content; smart cities such as Santander (Spain) and Assen (Netherlands) are beginning to wire themselves up with a large number of distributed sensors; and 100 hours of video (with sound) are uploaded you YouTube every minute.

However, the ability to understand and interact with all this sound data is hampered by a lack of tools allowing people to "make sense of sounds" based on the audio content. For example, in a sound map, users may be able to search for sound clips by geographical location, but not by "similar sounds". In broadcast archives, users must typically know which programme to look for, and listen through to find the section they need. Manually-entered textual metadata may allow text-based searching, but these typically only refer to the entire clip or programme, can often be ambiguous, and are hard to scale to large datasets. In addition, browsing sound data collections is a time-consuming process: without the help of e.g. key frame images available from video clips, each sound clip has to be "auditioned" (listened to) to find what is needed, and where the point of interest can be found. Radio programme producers currently have to train themselves to listen to audio clips at up to double speed to save time in the production process. Clearly better tools are needed.

To do this, we will investigate and develop new signal processing methods to analyse sound and audiovisual files, new interaction methods to search and browse through sets of sound files, and new methods to explore and understand the criteria searchers use when searching, selecting and interacting with sounds. The perceptual aspect will also investigate people's emotional response to sounds and soundscapes, assisting sound designers or producers to find audio samples with the effect they want to create, and informing the development of public policy on urban soundscapes and their impact on people.

There are a wide range of potential beneficiaries for the research and tools that will be produced in this project, including both professional users and the general public. Archivists who are digitizing content into sound and audiovisual archives will benefit from new ways to visualize and tag archive material. Radio or television programme makers will benefit from new ways to search through recorded programme material and databases of sound effects to reuse, and new tools to visualize and repurpose archive material once identified. Sound artists and musicians will benefit from new ways to find interesting sound objects, or collections of sounds, for them to use as part of compositions or installations. Educators will benefit from new ways to find material on particular topics (machines, wildlife) based on their sound properties rather than metadata. Urban planners and policy makers will benefit from new tools to understand the urban sound environment, and people living in those urban environments will benefit through improved city sound policies and better designed soundscapes, making the urban environment more pleasant. For the general public, many people are now building their own archives of recordings, in the form of videos with soundtracks, and may in future include photographs with associated sounds (audiophotographs). This research will help people make sense of the sounds that surround us, and the associations and memories that they bring.

Potential beneficiaries of this project outside of the academic research community include anyone who could benefit from new ways to explore sound and audiovisual data, or could benefit from access to the sounds that would be enabled by the research. Examples from different sectors are given below.

Commercial private sector:
* Commercial companies designing audio equipment, through easier access to new audio research;
* Musicians, composers and sound artists, through ways to find and explore new sounds as part of their creative output;
* Computer games companies, through new ways to reuse sound datasets creatively for new game sounds;
* Audio archiving companies, through access to the latest algorithms and methods for annotating and exploring sound archives;
* Television and radio companies, through ability to use sound data exploration technologies in the creation, editing and re-use of audio and audiovisual programmes.
* Acoustic consultants, through access to new ways of mapping and understanding soundscapes, which will help drive new design possibilities for the built environment.
* Internet-of-things companies who supply smart cities with networked sensor systems, through access to novel acoustic algorithms for more sophisticated mapping.
Policy-makers and others in government and government agencies:
* Urban planning authorities, through new insights into the impact of sounds and how to visualize and understand these impacts;
* Research funders, through establishment of a network of researchers in sound data research, opening up new opportunities for valuable research, and new demonstrators showing the value of research.

Public sector, third sector and others:
* Museums and other organizations with sound archives, through new software methods to allow people to explore and use their archives;
* Smart cities, through better ways to make sense of acoustic data from urban microphone arrays;
* Science promotion organizations, in particular through outputs from the projects on how people perceive and navigate sounds.

Wider public:
* People interested in exploring audio recordings at home, school, college or university, either for educational or general interest purposes;
* People recording sounds on mobiles and other portable devices, including those capturing audio as soundtracks to videos;
* Teachers in schools, colleges or universities who want to use sound examples for teaching audio or music;
* People living in urban environments, through improved city sound policies and better designed soundscapes, making the urban environment more pleasant;
* Audiences of creative output involving audio and music, through availability of new creative outputs facilitated by creative access to new sounds.

Researchers employed on the project:
* Improved skills in research methodologies, which may be transferred into e.g. the commercial private sector on completion of the project.