Cognitive and Structural Approaches to Contemporary Audiovisual Computer Aided Composition

Lead Research Organisation: Goldsmiths College
Department Name: Music

Abstract

Recent research in Neuroscience supports the long-held belief of many in the Visual arts and Experimental music community that there is a sensory connection between our experience of sound and our experience of light. These connections have been shown to be dynamic, changing in a reliable fashion depending on the way in which sonic and visual material is organised. The philosophical implications and potential cultural impact of this realisation is enormous. As such, it is carving a path for cutting edge research into the nature of human audiovisual perception in ways which have previously been thought impossible. Furthermore, recent progress into the development of real-time software for the simultaneous manipulation of sonic and visual material, in addition to the falling cost of increasingly more powerful multimedia technology, is making possible the exploration of multisensory, structurally related audiovisual composition techniques in ways which will revolutionise our perception of, and our interaction with moving images and sound.Concurrently, research in the fields of Cognition, Neuroscience, Psychology and Computation are highlighting specific new approaches to the structuring and perception of audiovisual material which may potentially be even more significant. This research reveals strategies for the understanding, engagement with and manipulation of our conscious experience of sound and vision. These strategies are reliable, repeatable, and demonstrative of the limits of consciousness itself. In the same way that the multisensory nature of audiovisual experience is already a feature of experimental audiovisual work, other extraordinary revelations in scientific and arts research have been and are becoming a major part of existing experimental audiovisual practice. These connections will only increase as research reveals evidence which aids in our understanding of how we interact with the universe. These facts alone support the specific engagement with contemporary theory and practice proposed by this research project.This research will engage and interact with leading academics in the fields of Neuroscience, Cognition, Psychology, Computation, Computer Music, Film and Visual arts. In addition it will liaise with contemporary Broadcast media practitioners from Television, Film and Computer Games design. It aims to discover and reveal novel concepts in the creation of audiovisual works using techniques uncovered through developments in the understanding of our engagement with audiovisual material. It will achieve this through collaboration, consultation, experimentation, software development, Audiovisual composition practice, Film making, Digital Media practice, Computer Music research, generative and algorithmic exploration and Artificial Intelligence research. This continuing program of interdisciplinary practice will be supported by the excellent research culture at Goldsmiths College. Housed in the Goldsmiths Electronic Music Studio, it will benefit from the direct involvement of institution's own researchers in the fields of Visual Perception, Music Cognition, Artificial Intelligence and Cognitive Neuropsychology. As such, the research will be taking place within an environment which will effectively support its successful development and completion.This research will result in the creation of specific works which exploit cognitive and structural approaches to audiovisual composition, and in addition gain high visibility at key international events which focus on contemporary approaches to Sound, Music, Film, Video and digital technology. It will engage with contemporary theory and practice at festivals, and also through publication. Furthermore, it will instantiate, develop and maintain a relevant discourse with the broadcast industries, and lead to an increased understanding of some of the most engaging and important issues surrounding our interaction with audiovisual and digital media culture.
 
Title Braindrop 
Description Brain-Computer interface based music composition that creates music based on subconscious signals detected by EEG signals 
Type Of Art Artistic/Creative Exhibition 
Year Produced 2009 
Impact International recogntion and Further funding as part of existing research outputs 
 
Title Christian Marclay, 'Clock', White Cube Gallery. 
Description I created the software and exhibition design for the Golden Lion award winning piece, "The Clock", by Christian Marclay, described as the best piece of media art ever. 
Type Of Art Artistic/Creative Exhibition 
Year Produced 2010 
Impact Continued worldwide exhibition, $500000 valuation. 
 
Title Delusions of Alien Control Part 4 
Description An interactive audiovisual installation 
Type Of Art Artistic/Creative Exhibition 
Year Produced 2008 
Impact I have been asked to perform this piece internationally on a number of occasions at festivals and events. 
 
Title Goldsmiths Postgraduate Live Electronic Music Performance Ensemble, 2007/8. 
Description A series of performances with Goldsmiths students as part of my fellowship 
Type Of Art Performance (Music, Dance, Drama, etc) 
Year Produced 2007 
Impact The continuation of the ensemble without my presence 
 
Title Study for Film and Audience 
Description Interactive art installation commissioned to mark the first instigation of the sound art category in the British Composer Awards 
Type Of Art Artistic/Creative Exhibition 
Year Produced 2008 
Impact First ever interactive exhibit as part of the British Council Composer Awards 
 
Description The worlds first Brain Computer Interface for Music that specifically allows for individual note choice using visually-evoked event-related potentials.

Algorithmic and AI-based music composition software for the creation of complex pieces of music based on brain-derived structural choices.

C++ software for use in Audiovisual Brain-Computer Interfacing between both medical and consumer grade Electroencephalographical equipment, and bespoke audiovisual composition environments.

Open source cross platform C++ synthesis library compatible with the iPhone, developed in order to simplify the learning process for making audiovisual art on any computer, including mobiles and smart phones.

Software tools for the deaf and hard of hearing based on cognitive visual and interaction approaches. Used in music therapy practice by a number of key practitioners and artists.

Open tutorials site for artists wishing to learn how to program contemporary audiovisual arts, including from a multisensory perspective (www.creativecomputation.co.uk).

Composition:

The pieces 'Delusions of Alien Control part 4' and 'Braindrop' were presented at a number of venues over the past two years.

Audiovisual Composition 'Study for Film and Audience' picked to represent the Sonic Arts category at the 2008 British Academy Composer Awards.
Exploitation Route Further successful research projects funded by the AHRC have been awarded as a result of the success of this project. Also, commercial software has been built that features outputs from this project.
Sectors Communities and Social Services/Policy,Creative Economy,Digital/Communication/Information Technologies (including Software),Education,Healthcare,Leisure Activities, including Sports, Recreation and Tourism,Culture, Heritage, Museums and Collections,Retail

URL http://www.mickgrierson.co.uk
 
Description The project was in many ways more successful than hoped. Following a considerable period of research into the neural multisensory basis of combined auditory-visual perception I gave several invited presentations internationally on the subject audiovisual relationships. The details of some of these are covered in my interim reports, but more recently I convened the Future of Light and Sound Symposium (March 2009), featuring international experts in this field of practice. Further to this, following my investigations into the practical applications of current thinking in Cognitive Theory, Neuroscience, Psychology, Artificial Intelligence, and Computation, I built the worlds first Brain Computer Interface for Music that specifically allows for individual note choice using visually-evoked event-related potentials in real-time. I then combined this approach with algorithmic and AI-based music composition to create complex pieces of music based on brain-derived structural choices. The Brain Computer Interface work was published at the International Computer Music Conference 2008, and the works were premiered later that year. In 2009, I created a number of C++ objects for use in Max/MSP/Jitter and Pure Data to facilitate Audiovisual Brain-Computer Interfacing between both medical and consumer grade Electroencephalographical equipment, and bespoke audiovisual composition environments. I have used this software to perform a number of concerts including at the 2009 Knowledge Futures Conference and the Barbican Lates in 2010. This software is also used internationally by a number of well known musicians and artists in recent Arts Council funded Brain-Music and visualisation related projects across a variety of styles and genres, including by jazz saxophonist Finn Peters as part of the 'Music of the Mind Project', which launches on Radio 4 and the Science Museum in September 2010. The software was also used by the widely-regarded Market Estates project in East London. In addition, in January 2010, I created an open source cross platform C++ synthesis library compatible with the iPhone. The library was developed in order to simplify the learning process for making audiovisual art on any computer, including mobiles and smart phones. It can be downloaded from www.maximilian.strangeloop.co.uk. In addition, I created software tools for the deaf and hard of hearing called 'LumiSonic', based on cognitive visual and interaction approaches. This is widely used in music therapy practice by a number of key practitioners and artists, notably Duncan Chapman. This project is ongoing but was already published and documented in my earlier interim report. It is currently distrubuted by national sound arts charity, soundandmusic.org, and is also available for the iPhone. Finally, a large, free, open tutorials site has been created for artists wishing to learn how to program contemporary audiovisual arts, including from a multisensory perspective. This can be accessed at www.creativecomputation.co.uk, and is housed by Goldsmiths College, being a primary source of information for both udergraduate and postgraduate programmes in creative computing related disciplines. Composition: The pieces 'Delusions of Alien Control part 4' and 'Braindrop' were presented at a number of venues over the past two years. These both utilised the above techniques, and were the first pieces to do so. Youtube videos of these and related studies/pieces total over 20,000 views. In addition, the piece, 'Study for Film and Audience' was picked to represent the Sonic Arts category at the 2008 British Academy Composer Awards. The wide dissemination of the project outputs has been praised by a number of AHRC colleagues, for which I am very grateful. This has led to new partnerships with brain-computer interface manufacturers GTEC and Neurosky, and more importantly games company Roll7 alongside whom I have successfully received 3 further years AHRC KTF funding.
First Year Of Impact 2008
Sector Communities and Social Services/Policy,Creative Economy,Digital/Communication/Information Technologies (including Software),Education,Healthcare,Leisure Activities, including Sports, Recreation and Tourism,Culture, Heritage, Museums and Collections,Retail
Impact Types Cultural,Societal,Economic,Policy & public services

 
Description Digital R&D
Amount £205,324 (GBP)
Organisation Nesta 
Sector Charity/Non Profit
Country United Kingdom
Start 04/2014 
End 11/2015
 
Description LumiSonic Project 
Organisation LumiSonic Project
Country United Kingdom 
Sector Charity/Non Profit 
PI Contribution I created software for the complex visualisation of sound and music textures specifically for deaf and disabled participants. This software is still distributed by the partner and used by teachers and project leaders working with deaf and disabled participants
Collaborator Contribution soundandmusic.org continue to distribute LumiSonic
Impact Further research as part of AHRC funded knowledge transfer fellowship
Start Year 2008
 
Description Music of the Mind', w/ Finn Peters, Arts Council of England. £20,000 project. 
Organisation Arts Council England
Country United Kingdom 
Sector Public 
PI Contribution Information taken from Final Report
 
Description Creative Programming MOOC 
Form Of Engagement Activity Engagement focused website, blog or social media channel
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Over 150,000 people enrolled on this MOOC, which was the world's first Massive Open Online Course in Creative Coding, and the first from any English University. It was a direct outcome from the Goldsmiths Creative Computing programme, which I have led since 2008. My AHRC funded Creative Fellowship directly impacted on the Creative Computing programme - one of the primary outputs was a website featuring the materials which ended up in this MOOC, and the fellowship led to my employment as programme leader. As a result, this MOOC is a direct output of my original AHRC funded Creative Fellowship. It is considered one of the top MOOCs of all time.
Year(s) Of Engagement Activity 2013,2014,2015
URL https://www.coursera.org/course/digitalmedia