Sound, Image and the Brain: Cognitive Live-Arts Technology in Contemporary Game-Oriented and Accessibility Paradigms.

Lead Research Organisation: Goldsmiths College
Department Name: Computing Department

Abstract

As a result of the ever-decreasing cost and increasing efficiency of contemporary consumer hardware and software, advanced audiovisual technologies are now commonplace, even in mobile telecommunications devices. Carefully designed products and services, such as the Netbook and the iPhone, exemplify modern transformations in the contemporary technoculture, providing not just new technology, but more importantly, new economic and social infrastructures by which we engage with audiovisual experience and beyond. These devices continue to offer fresh opportunities for and challenges to the way we perceive our everyday life, augmenting our understanding of the world around us, by delivering information to us at high speeds, and in increasingly intuitive ways.

This is to some extent enabled by developments in human computer interface research. Interface technologies that until now have been too expensive or too complex for mass production are being prototyped and deployed as part of new consumer devices. Amongst these devices are touch screen or 'surface' interfaces (of which the iPhone is the key example), miniature camera / projection based systems that deploy computer vision and machine listening, such as Microsoft's project Natal and MIT's Sixth Sense, and consumer grade brain-computer interfaces, such as Emotiv systems Epoch, and Neurosky's Mindset Electroencephalograph device.

These developments have captured the imagination of the public, perpetuating a narrative of the technological extension of humanity. Researchers exploring new directions for interface technology support this extension through terminology such as 'augmented reality' and 'sixth sense technology'. Importantly, what this revolution also brings is the capacity for technology to enhance the lives of those whose experience of everyday life is very different to that of the majority, augmenting sensory experience and capacity in an interesting and highly fertile way. Most significantly, rather than restricting the potential dissemination routes for the application of these technologies, collaboration with those whose sensory experience is different and/or unique continues to add huge value and impact to the refinement of these approaches, allowing lessons learned in less common situations to be deployed in more general scenarios, and vice versa. For example, brain-computer interface technology is of great potential value to those with limited mobility, cognitive impairment or other form of disability, and its use in these environments will continue to reveal the extent to which it is becoming generally applicable.

Through a multidisciplinary approach that draws on perception and cognition, media engineering, therapy, interactive gaming, sound, music and audiovisual arts, this proposal aims to take completed research in brain-computer interfaces, audio-visualisation, participation and gaming, and develop it in partnership with industry and public organisations in order that it might reach its full potential in terms of social and economic impact, by engaging more fully with those within the public sector who both stand to benefit from, and also contribute to the creation and enhancement of consumer-grade real-time interaction hardware and software for brain-computer interfacing and technology-led creativity.

Planned Impact

This proposed project has the potential for large public impact in both specific social, cultural and community arenas, and also in the mainstream economic sector. Given the main project partner is a limited company, and not a public organisation, its existence depends on continuing economic success. In addition, given the proposed Europe-wide exclusivity with respect to the Neurosky headset hardware, and that this device has only three technological competitors in the new marketplace, it is likely that this project will achieve high level public impact, not just in the UK, but all over the EU. As software developed through this project will be bundled directly with hardware (retailing for approx. £150 - £200), it will be well within the price range of ordinary consumers. In addition, any other party in Europe wishing to acquire and use the headset for any other purpose will receive bundled software developed as part of the project. This guarantees significant potential impact to a large number of potential stakeholders and observers alike. Furthermore, as Neurosky are a US company, economic and social impact is likely to extend to US territories also.

Crucially, as the project seeks to engage in the first instance with specific user groups through public sector partnership, the project stands to deliver significant social impact to the participating communities. We hope to be able to offer genuine improvements in the quality and experience of disabled user-group's engagement with sound, music, visual media and communications through effective real-world applications based on research in the electronic and technology arts sector. These improvements will be based on real-world user cases and testing, organised and ethically monitored by professional public bodies with years of expertise in the area of public engagement around technology arts in the disabled sectors. These workshops, to be organised in collaboration with qualified health professionals, will prove fine grained feedback on the responsiveness, ease of use and effectiveness of both visualisation based and EEG based therapeutic tools. These tools will then be given the maximum possible opportunity to impact upon the everyday lives and experience of the community user groups involved.

If successful, it is likely that this project will generate significant public attention in the mainstream press and even broadcast. This is supported by the already relatively high level public impact of the research project which featured on BBCi news (where it featured on the front page for several days), and BBC TV (including BBC's premiere TV program for the deaf and hard of hearing, 'See Hear', and 'The Gadget Show' (Roll7)), radio interviews including BBC World, BBC Radio 3, BBC World Service (radio), Russian TV, Discovery Channel Canada and Australian national radio among others. The work continues to be widely discussed on the Internet and in international press.

This demonstrates the size and importance of the potential genuine human interest in and around this project. Finally, it is expected that the combined social and economic impact of this proposed knowledge transfer fellowship has the capacity to dwarf the significant impact already represented by the research upon which it is based.

Publications

10 25 50

publication icon
Chapman D. (2014) N-gon waves - Audio applications of the geometry of regular polygons in the time domain in Proceedings - 40th International Computer Music Conference, ICMC 2014 and 11th Sound and Music Computing Conference, SMC 2014 - Music Technology Meets Philosophy: From Digital Echos to Virtual Ethos

publication icon
Grierson, M And Boon T (2013) Material Culture and Electronic Sound

publication icon
Jakubowski K (2015) The effect of exercise-induced arousal on chosen tempi for familiar melodies. in Psychonomic bulletin & review

publication icon
Mick Grierson (2011) Maximillian: An easy to use, cross platform C++ Toolkit for interactive audio and synthesis applications in Proceedings of the International Computer Music Conference

publication icon
Mital, P (2013) Mining Unlabeled Electronic Music Databases through 3D Interactive Visualization of Latent Component Relationships in Proceedings of the 2013 New Interfaces for Musical Expression Conference

 
Title Christian Marclay, 'Clock', White Cube Gallery. 
Description I created the software and exhibition design for the Golden Lion award winning piece, "The Clock", by Christian Marclay, described as the best piece of media art ever. 
Type Of Art Artistic/Creative Exhibition 
Year Produced 2010 
Impact Continued worldwide exhibition, $500000 valuation. 
 
Title Kurv Guitar 
Description Virtual Guitar product 
Type Of Art Artefact (including digital) 
Year Produced 2014 
Impact This has resulted in the creation of a new spin out company that has received significant funding 
URL https://www.kickstarter.com/projects/kurv/kurv-guitar
 
Title The Dean Rodney Singers 
Description I created 6 interactive music installations for the Heart n Soul project, 'The Dean Rodney Singers', an iPhone and iPad application for the Cultural Olympiad. It was the only interactive digital installation produced as part of the Paralympics unlimited festival at the South Bank Centre 
Type Of Art Artistic/Creative Exhibition 
Year Produced 2012 
Impact High impact digital arts exhibition as part of the cultural olympiad. 
 
Description This project is a knowledge transfer partnership that has been undertaken in partnership with commercial and public / third sector organisations in order to exploit existing research as part of a previous grant. In addition to this, wider questions regarding the applicability of practice-based arts research in audiovisual interaction, software development and accessibility in the wider digital economy. The results of this project are not yet fully public but it has featured prominently as part of the paralympics festival, and also has resulted in commissions from Sony entertainment for distribution on their handheld gaming console.
Exploitation Route The project has resulted in £150,000 worth of new consultancies as part of Creativeworks, and led to a further £500,000 in Innovate UK funding, and 2.2million in Horizon2020 funding. The software outputs of the fellowship are in approximately 20 products directly connected to these outcomes, and many more that are unconnected due to the software's open source nature. Some closed source aspects of the project are yet to be realised
Sectors Communities and Social Services/Policy,Creative Economy,Digital/Communication/Information Technologies (including Software),Education,Healthcare,Leisure Activities, including Sports, Recreation and Tourism,Culture, Heritage, Museums and Collections

URL http://www.mickgrierson.co.uk
 
Description The project led to direct participation in the Cultural Olympiad in the shape of the only interactive digital installation as part of the paralympics unlimited festival. This features 7 countries and 72 disabled musicians, participating to create 6 interactive music exhibits, iphone and ipad applications. The project has resulted in £150,000 worth of new consultancies as part of Creativeworks and beyond, and led to a further £500,000 in Innovate UK funding, and 2.2million in Horizon2020 funding. The software outputs of the fellowship are in approximately 20 products directly connected to these outcomes, and many more that are unconnected due to the software's open source nature. The project has also contributed to the development of new software via a now completed Horizon 2020 grant, RAPID-MIX, and a new AHRC grant on AI and music, MIMIC.
First Year Of Impact 2010
Sector Communities and Social Services/Policy,Creative Economy,Digital/Communication/Information Technologies (including Software),Education,Healthcare,Leisure Activities, including Sports, Recreation and Tourism,Culture, Heritage, Museums and Collections,Retail
Impact Types Cultural,Societal,Economic

 
Description AHRC Responsive Mode
Amount £1,008,366 (GBP)
Funding ID AH/R002657/1 
Organisation Arts & Humanities Research Council (AHRC) 
Sector Public
Country United Kingdom
Start 05/2018 
End 04/2021
 
Description Creativeworks / London Fusion
Amount £140,000 (GBP)
Organisation Creativeworks London 
Sector Charity/Non Profit
Country United Kingdom
Start 06/2013 
End 11/2014
 
Description Digital R&D
Amount £205,324 (GBP)
Organisation Nesta 
Sector Charity/Non Profit
Country United Kingdom
Start 04/2014 
End 11/2015
 
Description Horizon 2020
Amount € 2,400,000 (EUR)
Organisation European Commission 
Sector Public
Country European Union (EU)
Start 02/2015 
End 01/2018
 
Description TSB / Innovate UK Location based services
Amount £500,000 (GBP)
Organisation Innovate UK 
Sector Public
Country United Kingdom
Start 04/2015 
End 09/2016
 
Description Heart N Soul 
Organisation Heart N Soul
Country United Kingdom 
Sector Charity/Non Profit 
PI Contribution My team is the research partner on the Nesta/AHRC/ACE Digital R&D funded project, Soundlab
Collaborator Contribution Heart N Soul are the Collaborating partner on the soundlab project
Impact EU funded Horizon 2020 project, RAPID-MIX
Start Year 2013
 
Description Kurv Music Limited 
Organisation Kurv Music Limited
Country United Kingdom 
Sector Private 
PI Contribution I designed a new type of user interface for the company Kurv Music Limited.
Collaborator Contribution Prior art in the specific design
Impact The production of a new type of user interface
Start Year 2013
 
Description The Clock 
Organisation White Cube Gallery
Country United Kingdom 
Sector Private 
PI Contribution I designed the software and installation for Christian Marclay's critically acclaimed piece "The Clock", which has been called the greatest work of media art ever. The piece won the Golden Lion at the Venice Biennale. It is currently valued at approx $500,000
Collaborator Contribution White Cube financed and promoted the project
Impact Continuing worldwide exhibition
Start Year 2010
 
Description The Dean Rodney Singers 
Organisation Heart N Soul
Country United Kingdom 
Sector Charity/Non Profit 
PI Contribution I created 6 interactive music installations an iPhone and iPad app as part of the Paralympics Unlimited festival at the South Bank Centre. It was the only interactive digital installation that took place as part of the Paralympics related events
Collaborator Contribution Heart n Soul worked with Dean Rodney and 72 other artists from 7 countries to create an album of music content featuring disabled artists.
Impact NESTA Digital R&D funded project, "Soundlab". Creativeworks funded project, "The Fish Police"
Start Year 2012
 
Description The Oramics Machine Project 
Organisation Science Museum Group
Country United Kingdom 
Sector Charity/Non Profit 
PI Contribution In 2009, I recovered The Oramics Machine, the UK's first synthesiser, which was thought to be lost, and demonstrated that it featured a number of key innovations, including the world's first digitally controlled waveform generators - an innovation still necessary for all electronic and computer music instruments. I worked with Science Museum to bring the device and a range of collaborations around it to the public sphere through the exhibition "From Oramics to Electronica"
Collaborator Contribution The machine was sold to the Science Museum where it has been on special display ever since. It formed the basis of the exhibition "From Oramics to Electronica"
Impact AHRC Collaborative Doctoral Award - The Oramics Machine Exhibition - From Oramics to Electronica
Start Year 2009
 
Title EarWyrm 
Description Music game prototype developed to allow Roll7, an independent gaming company to pitch for products to SONY entertainment 
Type Of Technology Software 
Year Produced 2009 
Impact Roll7 were successful in being commissioned by SONY entertainment. 
 
Title Goldsmiths Digital 
Description A suite of software for conducting contract research and consultancy for the creative digital industries 
Type Of Technology Software 
Year Produced 2012 
Impact A range of successful consultancies generating income of approx. £150000 
 
Title Kurv 
Description I created a new musical interface as part of a partnership with Kurv Music. 
Type Of Technology Physical Model/Kit 
Year Produced 2015 
Impact This has led to the creation of a new music technology company, funded by a successful kickstarter campaign 
URL https://www.kickstarter.com/projects/kurv/kurv-guitar
 
Title Maximilian 
Description DSP library for interactive games, art and information retrieval 
Type Of Technology Software 
Year Produced 2009 
Impact the software is used by a wide range of artists and professionals as part of the openframeworks project 
URL http://maximilian.strangeloop.co.uk
 
Title Oramics Simulator 
Description An interactive Oramics machine simulator for the science museum's "Oramics" exhibition 
Type Of Technology Software 
Year Produced 2010 
Impact Has been described by the Science Museum as their "most popular interactive'. 
 
Title Spider Sonics 
Description Spider Sonics is an extension of LumiSonic that has been created as part of the Sound Image and Brain project in collaboration with a range of schools and colleges across the UK 
Type Of Technology Software 
Year Produced 2014 
Open Source License? Yes  
Impact As yet the software has only been tested across three schools but is looking great! 
 
Title The Dean Rodney Singers 
Description iPhone and iPad app as part of the cultural olympiad, co-designed by disabled musicians. 
Type Of Technology Software 
Year Produced 2012 
Open Source License? Yes  
Impact iPhone and iPad application co-designed by and for disabled musicians, released on the app store for free. 
 
Company Name Kurv Music 
Description The company manufactures and sells music technology 
Year Established 2015 
Impact This company has received significant investment from third-party investors and successful crowdfunding campaigns.
Website https://www.kickstarter.com/projects/kurv/kurv-guitar
 
Description AHRC Public Engagement Event 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Public/other audiences
Results and Impact I participated in the 2014 AHRC Creative Economy Showcase, with collaborating partners Heart n Soul and Roll7. Roll7 presented their game which was developed during the collaboration, which has since gone on to win a BAFTA
Year(s) Of Engagement Activity 2014
 
Description Sky News Interview 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Media (as a channel to the public)
Results and Impact I was interviewed by Sky news 'Swipe' as part of a special programme on new music technology.
Year(s) Of Engagement Activity 2016