Humanising Algorithmic Listening

Lead Research Organisation: University of Sussex
Department Name: Sch of Media, Film and Music

Abstract

Humanities researchers are opening their ears. From sound studies to oral history, media archaeology to ethnography, researchers are listening to the sonic environments of our lives, both off and online. In response to this digital, sonic turn, the Humanising Algorithmic Listening (HAL) network will bring together experts from across the humanities, sociology, machine listening, critical theory philosophy and interactive computer music to develop a research agenda for the design and development of computational methods for audio analysis - listening algorithms - in the arts and humanities.

Machine listening algorithms already exist but need tuning to align with and enrich humanities research. Existing methods are capable of tasks such as recognising melodies, identifying instruments or musical genres. Their power is evidenced by commercial IT products such as iPhone's Siri, which uses voice recognition software, or Spotify's recommender system that magically creates musical playlists based on our previous preferences. Driven by technical innovation and commercial interests, these tools are becoming at once more powerful, more complex and their inner workings more opaque. In order to support humanist research as tools for discovery, interpretation and insight, computational power must be accompanied by critical consideration of what it means to listen through algorithms, what new ways of listening might be afforded, and how we can learn to interact meaningfully with these black-boxed processes.

A rich philosophical literature examines the active role that technologies play in mediating our relationship with the world. These ideas have been considered in the context of interactive computer music, where software agents that listen and respond as musical partners have been created, but have yet to be thoroughly explored in the wider contexts of technologically-mediated listening. Developing a critical framework is crucial, firstly to underpin the design of new computational methods in arts and humanities - for example to support the interpretation and discovery of large audio archives and the design of new creative sound-based technologies. And secondly to deepen understanding of the role that these listening algorithms play in our everyday lives, thus contributing to current global digital ethics policy-making efforts.

In the course of three two-day meetings an agenda for algorithmic listening in the arts and humanities will be advanced. Workshop 1, led by PI Eldridge, will explore cultural questions and state-of-the art machine listening methods. Workshop 2, led by Co-I Stapleton, will focus on critical perspectives, exploring philosophical theories of human-machine interagency through interactive computer music systems. Workshop 3 will bring cultural, technical and critical perspectives together to propose an agenda for the co-design of Listening Algorithms that will stimulate future research and cultural understandings of listening algorithms.

Journal publications in each of the core areas will propagate discussions throughout relevant academic disciplines. Wider cultural impact will be achieved through a public panel in Brighton Digital Festival and feature on national radio. The network will disseminate its findings within and beyond academia through a strong, open access web-presence; it will be managed by a cross-sector Advisory Board of international experts to ensure relevance for and impact across relevant academic and public stakeholder communities.

This network builds on the University of Sussex recent £3 million investment in a Humanities Lab (SHL). It is led by Dr Alice Eldridge, a music practitioner and programmer experienced in applied machine listening, and co-led by Dr Paul Stapleton at Sonic Arts Research Centre (SARC), Queen's University Belfast. Both SHL and SARC have a proven track record in developing and delivering technically and critically challenging transdisciplinary research.

Planned Impact

By developing a research agenda for advancing critical and technical research in machine listening algorithms, the network activities will benefit a range of cultural third sector, commercial and creative IT industries and ultimately the formation of policies around the ethics of machine listening.

IA 1. New methods for managing and discovering audio archives in libraries and museums.
An upcoming challenge for all archives across all repositories is the handling of vast digital collections. New computational methods not only automate existing procedures but potentially afford novel means of discovery and navigation. Richard Ranft, Head of Sound and Vision at the British library and Fiona Courage of The Keep are on the Advisory Board to ensure that the network is informed by and informs the emerging needs of public audio archives.

IA 2. New application areas for machine listening and data science.
The sonic turn in the humanities, coupled with the exponential growth of audio archives represents a significant new field of application for machine listening and learning communities, within industry as well as academia. Methods for clustering and categorising music and bioacoustics data align within an engineering, problem solving framework; the types of questions and sonic data arising from humanities research are likely to challenge extant methods and drive novel perspectives. The network activities include a data challenge that will introduce DH data to the largest online community of international data scientists (see description of Kaggle Competition in letters of support).

IA 3. Enriched understanding of machine agencies for creative music technology designers and users.
Quasi-intelligent algorithms in commercial music technology products are increasingly common place. There have been large investments by companies such as Novation and Native Instruments (NI) in making controllers that algorithmically bolster interaction between humans and sound synthesis software, and substantial effort in EU funded projects such as Integra to support user understanding of such technology. This part of the music-industry sector is represented in the network by music technologist Tim Exile, creator of NI's voice controlled synthesizer, The Mouth. This network will enable NI and other music technology companies to draw on the expertise of academics. Ultimately, a philosophically informed design can support the development of richer new musical instruments, and to spur new collaborations between academics and commercial partners.


IA 4. Policy
Despite the increasing presence of Listening Algorithms (Spotify, Hello Barbie etc.) in everyday life, policy around the ethics of these and other automated algorithmic processes is in its infancy. The European Data Protection Supervisor's announcement in Jan 2016 of a new Digital Ethics advisory body promises to address how to ensure the integrity of our values while embracing the benefits of new technologies, but these policies are still in development. The network aims to generate discussion that could inform this nascent area of policy making. To this end, Leverhulme Centre for the Future of Intelligence member, Simon Goldhill is an Advisory Board member. Core participant Palle Dahlstedt links this network to another European initiative focused on global impact of algorithms http://thebiospherecode.com.

IA 5 Wider Public
The increasing ubiquity and fundamental opacity of listening algorithms in everyday devices means that enhancing public understanding of algorithmic listening, one of many technologies of automation which are rapidly populating our lives, is an important issue. Arts practices and surrounding debates have an important role in supporting public understandings of associated technical and ethical issues. Communities with artistic and/or digital media interests will be reached via festival and radio appearances outlined in Pathways to Impact.

Publications

10 25 50
 
Title 'New Communication' by Ens Ekt 
Description Ens Ekt (Paul Stapleton, Adam Pultz Melbye, Simon Rose) were commissioned by Initiative Neue Musik Berlin to create 'New Communication', a site-specific installation and series of performances in a WW2 communication bunker. This piece allowed the group to expand their research into material resonances and feedback networks shaped by machine listening algorithms into a large-scale dynamically evolving work that spanned the entire underground bunker. 
Type Of Art Performance (Music, Dance, Drama, etc) 
Year Produced 2019 
Impact Public audiences from broad demographics (e.g. ages ranged from 3 to 80+ years) interacted with the installation, which included the option for a guided tour to provide additional context about the network communication and machine listening technologies used in the work. Uptake in the tour was high, with many audiences demonstrating facination (through detailed questions) in the interactive potential of the installation along with the recontextualisation of the WW2 communicaiton bunker. Duration of audience stay averaged above 40min, indicating a high level of engagment with the work. 
URL https://vimeo.com/443162288
 
Title HAL Seed Project Oi! Algorithm, chew on this! 
Description John Bowers and Owen Green. All The Noises. 26 channel sound installation and performance-lecture with multiple interacting machine listening algorithms. Spatial Interaction Research Laboratory (SPIRAL), University of Huddersfield, Huddersfield, UK. 
Type Of Art Performance (Music, Dance, Drama, etc) 
Year Produced 2018 
Impact This work formed the basis of a NIME 2018 paper which was shortlisted for best conference paper, indicating a meaningful level of academic impact. 
 
Description The main aim of the network was to foster cross-sector collaboration and long-term impactful research that utilises and interrogates the use of computational methods in arts and humanities research and wider culture. The network ran for one year, included over 50 members across 10 countries and established a much needed critical space to grasp a whole range of heterodox approaches in machine listening, at a time when research in this area was predominantly technical. One of the central agreements was the need for constructive, critical and accessible information for wider communities of practice and publics, and a more nuanced use of machine listening in research and creative practice. Specific insights ranged from the value of interactive data perceptualisation to support interpretation of machine listening in scientific contexts to the recognition of machine listening as a creative material for composition.

Discussions and insights developed during the network have continued to evolve in subsequent years with network members now influential thinkers across European research projects, US industry (including facebook AI) and creative communities within and beyond academia.
Exploitation Route To enhance accessibility the website and you tube channel created during the network hosts over 30 talks by leading experts in a wide range of different disciplines concerned with machine listening. The value of this is evidence by the fact that the website itself is starting to be referenced in academic literature and receives visitors from all over the world.

Publications, code repositories, lecture series, teaching materials and performances emerged from the network and are all accessible via the website.
Sectors Creative Economy,Digital/Communication/Information Technologies (including Software),Environment,Culture, Heritage, Museums and Collections,Other

URL http://www.algorithmiclistening.org/
 
Description The main impacts to date have been in the cultural sector: The value of machine listening as compositional material has fed into ongoing research carried out by the PI and network member Dr Chris Kiefer into distributed machine listening in feedback systems. This has resulted in over 20 public concerts around Europe and four album releases (see https://www.feedbackcell.info/) The potential of machine listening in artistic contexts has also informed the work of the CI (Stapleton) and his research students (e.g. Pultz Melbye), in the creation of a commissioned installation and performances (Ens Ekt), as well as in the design of a new instrument (FAAB) which Pultz Melbye debuted in a performance at NIME 2020 that was awarded the conference's best music prize. Three-Body Problem (Stapleton, Pultz Melbye, and network member John Bowers), a new pandemic native trio formed in 2020, builds on approaches explored in HAL including: material resonance, distributed agency, machine listening and feedback musicianship. 3BP online performances to date include a lunchtime concert hosted at the Sonic Arts Research Centre, and a concert as part of NI Science Festival, with a forthcoming performance at New York City Electroacoustic Improvisation Summit.
First Year Of Impact 2018
Sector Culture, Heritage, Museums and Collections
Impact Types Cultural,Societal

 
Description INTERACT
Amount € 3,100 (EUR)
Funding ID WILDSENS 
Organisation INTERACT 
Sector Charity/Non Profit
Country Finland
Start 06/2018 
End 07/2018
 
Description FluCoMa 
Organisation University of Huddersfield
Country United Kingdom 
Sector Academic/University 
PI Contribution Following on from ideas discussed during the network grant I have been commissioned to create a new performance using machine listening tools under ERC grant FluCoMa (725899)
Collaborator Contribution The collaboration continues a research relationship with Owen Green who was a network member and is now PDRF on ERC project FluCoMa at the University of Huddersfield.
Impact In development (due 2021)
Start Year 2019
 
Title Application of Machine Listening in Oral History 
Description This is a set of python notebooks introducing core machine listening methods to oral historians. The methods are not new in themselves, but constitute a knowledge transfer from one discipline to another. 
Type Of Technology Webtool/Application 
Year Produced 2017 
Open Source License? Yes  
Impact The notebooks were created for two workshops: One at Digital Humanities Conference 2016: Music Information Retrieval Algorithms for Oral History Collections (http://dh2016.adho.org/abstracts/10) 
 
Title Oi Algorithm repo 
Description This project was a collaboration between John Bowers and Owen Green, under the umbrella of a research network, Humanising Algorithmic Listening, run by Alice Eldridge and Paul Stapleton. This repo provides a home for the various Max patches I produced along the way, as we went through a process of 'many makes', intended to probe, playfully, the idea and place of machine listening in music. A blog post with more information, or a NIME paper with both more information, more chutzpah, and more situationism. 
Type Of Technology Software 
Year Produced 2018 
Open Source License? Yes  
Impact The approach developed here articulates a curious, critical approach to machine listening as opposed to un-politicised mainstream of 'purely' technical research. 
 
Description DIY radio transmission event 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact Two workshops of 14 people each were run at a local art gallery (ONCA). The workshops were led by a Soundtent - an artist run organisation - and introduced audiences to DIY electronics, giving an overview of how to make a radio transmitter from a raspberry pi. Audiences then had access to all necessary materials. The workshop sparked a couple of future collaborations between members. They workshop cemented links between the university, the art gallery ONCA, the South Downs Trust, various local media groups (Brighton Digital Festival) and the Universities of Sussex and Brighton music and media arts courses.
Year(s) Of Engagement Activity 2017
URL http://brightondigitalfestival.co.uk/events/diy-radio-transmission-pi-streambox-workshop
 
Description Electric Spring 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact 40 people attended a talk I gave at Electric Spring entitled "Ecosystemics & Ecoacoustics Creative coding for speculative faction & (re)sensorising empiricism"
My approach changed the thinking of many audience members. I have since had emails from the students of audience members wishing to interview me for their research projects.
Year(s) Of Engagement Activity 2020
URL http://electricspring.co.uk/#ccl-symposium
 
Description Long Table @ Brighton Digital Festival 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact Humanising Algorithmic Listening in Culture and Conservation at Brighton Digital Festival. Network members hosted a public debate in an informal, semi-theatrical setting of the Rose Hill Arts centre. Audience members were largely artists and musicians interested in the implications of new technologies on daily life. Many reported knowing very little about the introduction of listening algorithms in every day technologies. The open format discussion was a real success. Following the success of this event the university plans further open format research talks at this venue.
Year(s) Of Engagement Activity 2017
URL http://brightondigitalfestival.co.uk/events/humanising-algorithmic-listening-culture-conservation
 
Description TECHNE workshop 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Postgraduate students
Results and Impact 40ish TECHNE students attended a day long workshop at CRiSAP, UAL. All were arts and humanities students new to coding. A wide range of responses from revulsion at workignwith maths to pure delight at being able to listen in new ways. Vast majority reported major new perspective on approaching working with oral archives
Year(s) Of Engagement Activity 2017
URL http://www.techne.ac.uk/for-students/techne-events/apr-2015/data-mining-the-audio-of-oral-history-a-...