Augmented Sonic Reality

Lead Research Organisation: University of Nottingham
Department Name: School of Computer Science

Abstract

The sounds and noises of our urban landscapes provide the artist with a fertile resource with which to reflect society back at itself. By listening to these noises with greater scrutiny we can gain a better understanding of our environment and insights into possible futures.

In addition, urban landscapes are awash with hidden noises, such as network and communication data, subsonic and ultrasonic frequencies, radio signals, electromagnetic fields and the transitory data noise of its inhabitants.

This PhD will involve artist-led, and practice-based research and development into ways in which listeners can create personal, interactive and immersive mobile sound art experiences, or interactive soundscapes. It is proposed that these interactive soundscapes could enhance perceptions of interior and exterior urban environments and provide insights into, and a more accurate document of, our contemporary urban landscapes.

Detailed research will be undertaken into ways of effectively capturing and processing environmental and personal data sources for the purpose of its sonification and inclusion within the soundscape.

It is also envisaged that this project will involve the research and development of a wearable audio and data capturing interface that would facilitate an accessible and mobile audio experience. Such an interface could incorporate binaural recording and monitoring, along with environmental sensing and smartphone technologies.

Increasing the granularity of the data and the fidelity of the audio captured would build significantly on previous work in this area, expanding options relating to data mapping, processing and sonification, and resulting in a more detailed soundscape.

The personal sound art creation and performance experience that this project potentially offers, also offers exciting opportunities for the generation of new musical encounters. The empowerment of the listener as performer and the embracing of ambient noise are considered key compositional elements in the realisation of such new musical experiences.

Planned Impact

We intend the Horizon CDT to be the place where partners come to find their future employees and to engage with the opportunities and challenges of digital identity and personal data. The key beneficiaries of our research will be:

- Commercial private sector companies that will engage with our CDT students during their research and/or employ them after graduation. Our partners include companies developing digital identity technologies as well as user companies across a range of sectors (consumer goods, entertainment, transportation, energy and others).

- Public sector and third sector organisations that are concerned with the use of digital identities to support civil society including broadcasters, healthcare providers and campaign groups.

- The public whose personal data forms the focus of their research and who will ultimately use and come to depend upon digital identities.

- Research communities spanning computer science, engineering, psychology, sociology, business and humanities.

These will benefit in various ways.

- Commercial, public and third sector companies will benefit from being able to recruit from a pool of talented PhD graduates who bring an in-depth understanding of digital identity and a proven ability to work in interdisciplinary teams. They will also benefit from being able to participate in co-creation of PhD research to ensure focus on relevant challenges and be able to exploit results of this PhD research.

- The public will benefit through a greater understanding of the opportunities and challenges of digital identity.

- Research communities will benefit by opening up promising new interdisciplinary fields.

Our Impact activities will be driven by Professor Derek McAuley, the Director of Horizon, who has a track record of establishing industry labs, spinning our companies and who is currently acting CIO of the TSB funded Connected Digital Economy Hub. Key activities will be:

- All Horizon PhDs will be carried out in collaboration with an external partner who will be involved in drawing up the initial topic, recruiting students, shaping the PhD proposal, supervision, and hosting at least one internship

- We will continue to organise knowledge exchange events within Horizon that are open to our network of over 100 external partners, including our annual Horizon Research Conference.

- We will encourage the release of applications, open source software, and open datasets wherever collaboration agreements allow.

- We will actively encourage our students to spin-out new ventures, including providing seedcorn funding through Horizon.

- We will engage our students with our two partner catapults, the Connected Digital Economy Catapult and the Satellite Applications Catapult.

- We will actively encourage industry visits through guest lectures on our "Broadening Horizons" core taught programme.

- We will also encourage companies to define, steer and sponsor the first year interdisciplinary team projects.

- Our students will complete a module on Public and External Engagement and are encouraged to engage in public events and exhibitions.

- Horizon's journalist-in-residence will help expose students research to the wider world through regular blogposts, while the University's marketing and communications team will help them develop press releases.

- We will provide training in research publication as part of the Professional Skills module and mentor publications through the Practice Led Project and the annual writing retreat (where students present and critique draft papers).

These impact activities will be supported by a professional online presence with posters, demos and podcasts made available through our website and associated YouTube channel and twitter feed, and with individual PhD profiles being posted on our own site and on external networking portals such as LinkedIn and ResearchGate.

Publications

10 25 50
publication icon
Cliffe L (2019) The Audible Artefact

publication icon
Cliffe L (2021) Materialising contexts: virtual soundscapes for real-world exploration. in Personal and ubiquitous computing

 
Title Horror-Fi Me 
Description Horror-Fi Me is an audio augmented reality (AAR) iOS mobile application. It enables users to author and experience an AAR environment from within any indoor location. It enables the creation of an explorable and interactive cinematic soundscape. 
Type Of Art Artefact (including digital) 
Year Produced 2020 
Impact As of November 2020, this mobile application has logged over 60 downloads and installations. 
URL https://horror-fi.me
 
Description An augmented reality software application has been developed, tested and analysed which demonstrates the potential for increasing visitor engagement with physical exhibition content and digital audio archival content by realising audio augmented objects. Audio augmented objects are physical objects that have had virtual audio content attached to them, and a collection of audio augmented objects can realise a virtual and physically explorable soundscape. Furthermore, audio augmented objects demonstrate themselves as useful additions to audio augmented reality experiences by helping to ground virtual audio content in physical reality and realising a heightened augmented reality experience.
Exploitation Route Insights, software and best practices could potentially be utilised by curatorial and creative practitioners to increase visitor exploration of, and engagement with, museum, gallery and heritage sites, their existing collections, spaces and artefacts therein. As well as providing potential interfaces to digital audio archive material.
Sectors Creative Economy,Culture, Heritage, Museums and Collections

URL https://doi.org/10.1007/s00779-020-01405-3
 
Description Two public-facing sound art installations and a publicly available augmented reality mobile application have so far been realised using the findings and software developed as a part of this research.
First Year Of Impact 2020
Sector Culture, Heritage, Museums and Collections
Impact Types Cultural

 
Description Audible Artefacts 
Organisation Science Museum Group
Country United Kingdom 
Sector Public 
PI Contribution Creative and research based collaboration between MRL (University of Nottingham), Department of Culture and visual studies (University of Nottingham) and Nation Science and Media Museum (The Science Museum Group). This collaboration contributed to the research and development of an Audio Augmented Reality Mobile Application, a related visitor focus group study, a public sited sound installation, and a technical demonstration and presentation.
Collaborator Contribution The Science Museum Group supported the public engagement activities by hosting the related public engagement events, providing access to museum artefacts and curatorial and collections staff. UoN Department of Culture also provided assistance with public engagement activities.
Impact Science Museum Listening Session, Public and professional engagement/focus group demonstration, Public sound installation.
Start Year 2019
 
Description Psychical Laboratory Sound Installation 
Organisation University of Stirling
Country United Kingdom 
Sector Academic/University 
PI Contribution Design and implementation of an audio augmented reality sound installation.
Collaborator Contribution The partners provided a platform and environment (a sound installation at a cultural festival event) to publicly deploy a recently developed interactive and immersive mobile audio application experience.
Impact This was a multi-disciplinary project across Computer Science (mobile and interactive audio application development) and cultural studies (researching and provision of audio archival content for use within the developed application).
Start Year 2021
 
Title SoundScaper 
Description A smartphone mobile application to situate virtual sound sources within physical space, through which participants can interact within an Audio Augmented Reality (AAR) experience, and with which curatorial and artistic practitioners can also author their own experiences. 
Type Of Technology Webtool/Application 
Year Produced 2019 
Impact It was used in two public engagement experiences at Nation Science & Media Museum, Bradford. 
 
Description DMRN+ 12: Digital Music Research Network One-day Workshop 2017 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact Poster presentation at a digital music workshop, discussed ideas with both national and international academics and industry practitioners.
Year(s) Of Engagement Activity 2017
URL https://www.researchgate.net/profile/Panos_Kudumakis/publication/323942840_DMRN12_Digital_Music_Rese...
 
Description Listening Session 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Study participants or study members
Results and Impact Focus group and study workshop on virtual soundscapes and Audio Augmented Reality held at the Science and Media Museum, Bradford, UK. Participation in the study prompted discussions around the technology's effective deployment within cultural spaces amongst participants.
Year(s) Of Engagement Activity 2019
 
Description Presentation given at Live Cinema III Conference 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact A presentation given on the subject of audio augmented reality at Live Cinema III conference. The presentation was accompanied by a demonstrative mobile application, which was made available for attendees to try prior to and after the conference event.
Year(s) Of Engagement Activity 2020
URL http://livecinema.org.uk/live-cinema-conference-iii-the-reopening/
 
Description Science Museum Blog Article 
Form Of Engagement Activity Engagement focused website, blog or social media channel
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact Blog post for the Science and Media Museum website (guest author) about how the design of analogue music equipment influenced the online interactive experiments in the museum's Sonic Futures project.
Year(s) Of Engagement Activity 2019
URL https://blog.scienceandmediamuseum.org.uk/listening-to-digital-dust/
 
Description Science Museum Lates 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact Public sound installation at National Science & Media Museum, Bradford. Experienced by 45 members of the public within the context of a free, open invitation, late night open gallery/museum event. The overall event was designed to engage public audiences with new research and technologies, the installation prompted discussions with many participants around the subject of Audio Augmented Reality.
Year(s) Of Engagement Activity 2019
 
Description Talk and presentation given to Society of British Theatre Designers (SBTD) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact A talk delivered to approx 20 members of Society of British Theatre Designers (SBTD) on the subject (and recent research findings) relating to audio augmented reality.
Year(s) Of Engagement Activity 2020
URL http://www.theatredesign.org.uk/news-story/launch-of-the-sbtd-new-collaborations-working-group/
 
Description TechFast Talk 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Industry/Business
Results and Impact Morning presentation and subsequent discussion around the subject of Audio Augmented Reality (AAR) and presentation of recent research findings around virtual soundscape authoring and production. Presentation took place at a local design consultancy business, and was attended by their employees, along with others from other local businesses. Audience discussion was largely around possible commercial applications of the research presented.
Year(s) Of Engagement Activity 2020
URL https://www.meetup.com/Notts-Techfast/events/268502129/