Sound, Image and the Brain: Cognitive Live-Arts Technology in Contemporary Game-Oriented and Accessibility Paradigms.
Lead Research Organisation:
Goldsmiths University of London
Department Name: Computing Department
Abstract
As a result of the ever-decreasing cost and increasing efficiency of contemporary consumer hardware and software, advanced audiovisual technologies are now commonplace, even in mobile telecommunications devices. Carefully designed products and services, such as the Netbook and the iPhone, exemplify modern transformations in the contemporary technoculture, providing not just new technology, but more importantly, new economic and social infrastructures by which we engage with audiovisual experience and beyond. These devices continue to offer fresh opportunities for and challenges to the way we perceive our everyday life, augmenting our understanding of the world around us, by delivering information to us at high speeds, and in increasingly intuitive ways.
This is to some extent enabled by developments in human computer interface research. Interface technologies that until now have been too expensive or too complex for mass production are being prototyped and deployed as part of new consumer devices. Amongst these devices are touch screen or 'surface' interfaces (of which the iPhone is the key example), miniature camera / projection based systems that deploy computer vision and machine listening, such as Microsoft's project Natal and MIT's Sixth Sense, and consumer grade brain-computer interfaces, such as Emotiv systems Epoch, and Neurosky's Mindset Electroencephalograph device.
These developments have captured the imagination of the public, perpetuating a narrative of the technological extension of humanity. Researchers exploring new directions for interface technology support this extension through terminology such as 'augmented reality' and 'sixth sense technology'. Importantly, what this revolution also brings is the capacity for technology to enhance the lives of those whose experience of everyday life is very different to that of the majority, augmenting sensory experience and capacity in an interesting and highly fertile way. Most significantly, rather than restricting the potential dissemination routes for the application of these technologies, collaboration with those whose sensory experience is different and/or unique continues to add huge value and impact to the refinement of these approaches, allowing lessons learned in less common situations to be deployed in more general scenarios, and vice versa. For example, brain-computer interface technology is of great potential value to those with limited mobility, cognitive impairment or other form of disability, and its use in these environments will continue to reveal the extent to which it is becoming generally applicable.
Through a multidisciplinary approach that draws on perception and cognition, media engineering, therapy, interactive gaming, sound, music and audiovisual arts, this proposal aims to take completed research in brain-computer interfaces, audio-visualisation, participation and gaming, and develop it in partnership with industry and public organisations in order that it might reach its full potential in terms of social and economic impact, by engaging more fully with those within the public sector who both stand to benefit from, and also contribute to the creation and enhancement of consumer-grade real-time interaction hardware and software for brain-computer interfacing and technology-led creativity.
This is to some extent enabled by developments in human computer interface research. Interface technologies that until now have been too expensive or too complex for mass production are being prototyped and deployed as part of new consumer devices. Amongst these devices are touch screen or 'surface' interfaces (of which the iPhone is the key example), miniature camera / projection based systems that deploy computer vision and machine listening, such as Microsoft's project Natal and MIT's Sixth Sense, and consumer grade brain-computer interfaces, such as Emotiv systems Epoch, and Neurosky's Mindset Electroencephalograph device.
These developments have captured the imagination of the public, perpetuating a narrative of the technological extension of humanity. Researchers exploring new directions for interface technology support this extension through terminology such as 'augmented reality' and 'sixth sense technology'. Importantly, what this revolution also brings is the capacity for technology to enhance the lives of those whose experience of everyday life is very different to that of the majority, augmenting sensory experience and capacity in an interesting and highly fertile way. Most significantly, rather than restricting the potential dissemination routes for the application of these technologies, collaboration with those whose sensory experience is different and/or unique continues to add huge value and impact to the refinement of these approaches, allowing lessons learned in less common situations to be deployed in more general scenarios, and vice versa. For example, brain-computer interface technology is of great potential value to those with limited mobility, cognitive impairment or other form of disability, and its use in these environments will continue to reveal the extent to which it is becoming generally applicable.
Through a multidisciplinary approach that draws on perception and cognition, media engineering, therapy, interactive gaming, sound, music and audiovisual arts, this proposal aims to take completed research in brain-computer interfaces, audio-visualisation, participation and gaming, and develop it in partnership with industry and public organisations in order that it might reach its full potential in terms of social and economic impact, by engaging more fully with those within the public sector who both stand to benefit from, and also contribute to the creation and enhancement of consumer-grade real-time interaction hardware and software for brain-computer interfacing and technology-led creativity.
Planned Impact
This proposed project has the potential for large public impact in both specific social, cultural and community arenas, and also in the mainstream economic sector. Given the main project partner is a limited company, and not a public organisation, its existence depends on continuing economic success. In addition, given the proposed Europe-wide exclusivity with respect to the Neurosky headset hardware, and that this device has only three technological competitors in the new marketplace, it is likely that this project will achieve high level public impact, not just in the UK, but all over the EU. As software developed through this project will be bundled directly with hardware (retailing for approx. £150 - £200), it will be well within the price range of ordinary consumers. In addition, any other party in Europe wishing to acquire and use the headset for any other purpose will receive bundled software developed as part of the project. This guarantees significant potential impact to a large number of potential stakeholders and observers alike. Furthermore, as Neurosky are a US company, economic and social impact is likely to extend to US territories also.
Crucially, as the project seeks to engage in the first instance with specific user groups through public sector partnership, the project stands to deliver significant social impact to the participating communities. We hope to be able to offer genuine improvements in the quality and experience of disabled user-group's engagement with sound, music, visual media and communications through effective real-world applications based on research in the electronic and technology arts sector. These improvements will be based on real-world user cases and testing, organised and ethically monitored by professional public bodies with years of expertise in the area of public engagement around technology arts in the disabled sectors. These workshops, to be organised in collaboration with qualified health professionals, will provide fine grained feedback on the responsiveness, ease of use and effectiveness of both visualisation based and EEG based therapeutic tools. These tools will then be given the maximum possible opportunity to impact upon the everyday lives and experience of the community user groups involved.
If successful, it is likely that this project will generate significant public attention in the mainstream press and even broadcast. This is supported by the already relatively high level public impact of the research project which featured on BBCi news (where it featured on the front page for several days), and BBC TV (including BBC's premiere TV program for the deaf and hard of hearing, 'See Hear', and 'The Gadget Show' (Roll7)), radio interviews including BBC World, BBC Radio 3, BBC World Service (radio), Russian TV, Discovery Channel Canada and Australian national radio among others. The work continues to be widely discussed on the Internet and in international press.
This demonstrates the size and importance of the potential genuine human interest in and around this project. Finally, it is expected that the combined social and economic impact of this proposed knowledge transfer fellowship has the capacity to dwarf the significant impact already represented by the research upon which it is based.
Crucially, as the project seeks to engage in the first instance with specific user groups through public sector partnership, the project stands to deliver significant social impact to the participating communities. We hope to be able to offer genuine improvements in the quality and experience of disabled user-group's engagement with sound, music, visual media and communications through effective real-world applications based on research in the electronic and technology arts sector. These improvements will be based on real-world user cases and testing, organised and ethically monitored by professional public bodies with years of expertise in the area of public engagement around technology arts in the disabled sectors. These workshops, to be organised in collaboration with qualified health professionals, will provide fine grained feedback on the responsiveness, ease of use and effectiveness of both visualisation based and EEG based therapeutic tools. These tools will then be given the maximum possible opportunity to impact upon the everyday lives and experience of the community user groups involved.
If successful, it is likely that this project will generate significant public attention in the mainstream press and even broadcast. This is supported by the already relatively high level public impact of the research project which featured on BBCi news (where it featured on the front page for several days), and BBC TV (including BBC's premiere TV program for the deaf and hard of hearing, 'See Hear', and 'The Gadget Show' (Roll7)), radio interviews including BBC World, BBC Radio 3, BBC World Service (radio), Russian TV, Discovery Channel Canada and Australian national radio among others. The work continues to be widely discussed on the Internet and in international press.
This demonstrates the size and importance of the potential genuine human interest in and around this project. Finally, it is expected that the combined social and economic impact of this proposed knowledge transfer fellowship has the capacity to dwarf the significant impact already represented by the research upon which it is based.
People |
ORCID iD |
Mick Grierson (Principal Investigator) |
Publications
Broad T
(2017)
Autoencoding Blade Runner : Reconstructing Films with Artificial Neural Networks
in Leonardo
Williamson VJ
(2012)
Perception and action de-coupling in congenital amusia: sensitivity to task demands.
in Neuropsychologia
Mick Grierson
(2011)
Progress Report on the EAVI BCI Toolkit for Music: Musical Applications of Algorithms for use with consumer brain computer interfaces
in Proc. ICMC, Huddersfield
Chapman D.
(2014)
N-gon waves - Audio applications of the geometry of regular polygons in the time domain
in Proceedings - 40th International Computer Music Conference, ICMC 2014 and 11th Sound and Music Computing Conference, SMC 2014 - Music Technology Meets Philosophy: From Digital Echos to Virtual Ethos
Mital, P
(2013)
Mining Unlabeled Electronic Music Databases through 3D Interactive Visualization of Latent Component Relationships
in Proceedings of the 2013 New Interfaces for Musical Expression Conference
Mick Grierson
(2011)
Maximillian: An easy to use, cross platform C++ Toolkit for interactive audio and synthesis applications
in Proceedings of the International Computer Music Conference
Jakubowski K
(2015)
The effect of exercise-induced arousal on chosen tempi for familiar melodies.
in Psychonomic bulletin & review
Mital P
(2013)
Corpus-based visual synthesis
Title | Christian Marclay, 'Clock', White Cube Gallery. |
Description | I created the software and exhibition design for the Golden Lion award winning piece, "The Clock", by Christian Marclay, described as the best piece of media art ever. |
Type Of Art | Artistic/Creative Exhibition |
Year Produced | 2010 |
Impact | Continued worldwide exhibition, $500000 valuation. |
Title | Kurv Guitar |
Description | Virtual Guitar product |
Type Of Art | Artefact (including digital) |
Year Produced | 2014 |
Impact | This has resulted in the creation of a new spin out company that has received significant funding |
URL | https://www.kickstarter.com/projects/kurv/kurv-guitar |
Title | The Dean Rodney Singers |
Description | I created 6 interactive music installations for the Heart n Soul project, 'The Dean Rodney Singers', an iPhone and iPad application for the Cultural Olympiad. It was the only interactive digital installation produced as part of the Paralympics unlimited festival at the South Bank Centre |
Type Of Art | Artistic/Creative Exhibition |
Year Produced | 2012 |
Impact | High impact digital arts exhibition as part of the cultural olympiad. |
Description | This project is a knowledge transfer partnership that has been undertaken in partnership with commercial and public / third sector organisations in order to exploit existing research as part of a previous grant. In addition to this, wider questions regarding the applicability of practice-based arts research in audiovisual interaction, software development and accessibility in the wider digital economy. The results of this project are not yet fully public but it has featured prominently as part of the paralympics festival, and also has resulted in commissions from Sony entertainment for distribution on their handheld gaming console. |
Exploitation Route | The project has resulted in £150,000 worth of new consultancies as part of Creativeworks, and led to a further £500,000 in Innovate UK funding, and 2.2million in Horizon2020 funding. The software outputs of the fellowship are in approximately 20 products directly connected to these outcomes, and many more that are unconnected due to the software's open source nature. Some closed source aspects of the project are yet to be realised |
Sectors | Communities and Social Services/Policy Creative Economy Digital/Communication/Information Technologies (including Software) Education Healthcare Leisure Activities including Sports Recreation and Tourism Culture Heritage Museums and Collections |
URL | http://www.mickgrierson.co.uk |
Description | The project led to direct participation in the Cultural Olympiad in the shape of the only interactive digital installation as part of the paralympics unlimited festival. This features 7 countries and 72 disabled musicians, participating to create 6 interactive music exhibits, iphone and ipad applications. The project has resulted in £150,000 worth of new consultancies as part of Creativeworks and beyond, and led to a further £500,000 in Innovate UK funding, and 2.2million in Horizon2020 funding. The software outputs of the fellowship are in approximately 20 products directly connected to these outcomes, and many more that are unconnected due to the software's open source nature. The project has also contributed to the development of new software via a now completed Horizon 2020 grant, RAPID-MIX, and a new AHRC grant on AI and music, MIMIC. |
First Year Of Impact | 2010 |
Sector | Communities and Social Services/Policy,Creative Economy,Digital/Communication/Information Technologies (including Software),Education,Healthcare,Leisure Activities, including Sports, Recreation and Tourism,Culture, Heritage, Museums and Collections,Retail |
Impact Types | Cultural Societal Economic |
Description | AHRC Responsive Mode |
Amount | £1,008,366 (GBP) |
Funding ID | AH/R002657/1 |
Organisation | Arts & Humanities Research Council (AHRC) |
Sector | Public |
Country | United Kingdom |
Start | 04/2018 |
End | 04/2021 |
Description | Creativeworks / London Fusion |
Amount | £140,000 (GBP) |
Organisation | Creativeworks London |
Sector | Charity/Non Profit |
Country | United Kingdom |
Start | 05/2013 |
End | 11/2014 |
Description | Digital R&D |
Amount | £205,324 (GBP) |
Organisation | Nesta |
Sector | Charity/Non Profit |
Country | United Kingdom |
Start | 03/2014 |
End | 11/2015 |
Description | Horizon 2020 |
Amount | € 2,400,000 (EUR) |
Organisation | European Commission |
Sector | Public |
Country | European Union (EU) |
Start | 02/2015 |
End | 01/2018 |
Description | TSB / Innovate UK Location based services |
Amount | £500,000 (GBP) |
Organisation | Innovate UK |
Sector | Public |
Country | United Kingdom |
Start | 03/2015 |
End | 09/2016 |
Description | Heart N Soul |
Organisation | Heart N Soul |
Country | United Kingdom |
Sector | Charity/Non Profit |
PI Contribution | My team is the research partner on the Nesta/AHRC/ACE Digital R&D funded project, Soundlab |
Collaborator Contribution | Heart N Soul are the Collaborating partner on the soundlab project |
Impact | EU funded Horizon 2020 project, RAPID-MIX |
Start Year | 2013 |
Description | Kurv Music Limited |
Organisation | Kurv Music Limited |
Country | United Kingdom |
Sector | Private |
PI Contribution | I designed a new type of user interface for the company Kurv Music Limited. |
Collaborator Contribution | Prior art in the specific design |
Impact | The production of a new type of user interface |
Start Year | 2013 |
Description | The Clock |
Organisation | White Cube Gallery |
Country | United Kingdom |
Sector | Private |
PI Contribution | I designed the software and installation for Christian Marclay's critically acclaimed piece "The Clock", which has been called the greatest work of media art ever. The piece won the Golden Lion at the Venice Biennale. It is currently valued at approx $500,000 |
Collaborator Contribution | White Cube financed and promoted the project |
Impact | Continuing worldwide exhibition |
Start Year | 2010 |
Description | The Dean Rodney Singers |
Organisation | Heart N Soul |
Country | United Kingdom |
Sector | Charity/Non Profit |
PI Contribution | I created 6 interactive music installations an iPhone and iPad app as part of the Paralympics Unlimited festival at the South Bank Centre. It was the only interactive digital installation that took place as part of the Paralympics related events |
Collaborator Contribution | Heart n Soul worked with Dean Rodney and 72 other artists from 7 countries to create an album of music content featuring disabled artists. |
Impact | NESTA Digital R&D funded project, "Soundlab". Creativeworks funded project, "The Fish Police" |
Start Year | 2012 |
Description | The Oramics Machine Project |
Organisation | Science Museum Group |
Country | United Kingdom |
Sector | Public |
PI Contribution | In 2009, I recovered The Oramics Machine, the UK's first synthesiser, which was thought to be lost, and demonstrated that it featured a number of key innovations, including the world's first digitally controlled waveform generators - an innovation still necessary for all electronic and computer music instruments. I worked with Science Museum to bring the device and a range of collaborations around it to the public sphere through the exhibition "From Oramics to Electronica" |
Collaborator Contribution | The machine was sold to the Science Museum where it has been on special display ever since. It formed the basis of the exhibition "From Oramics to Electronica" |
Impact | AHRC Collaborative Doctoral Award - The Oramics Machine Exhibition - From Oramics to Electronica |
Start Year | 2009 |
Title | EarWyrm |
Description | Music game prototype developed to allow Roll7, an independent gaming company to pitch for products to SONY entertainment |
Type Of Technology | Software |
Year Produced | 2009 |
Impact | Roll7 were successful in being commissioned by SONY entertainment. |
Title | Goldsmiths Digital |
Description | A suite of software for conducting contract research and consultancy for the creative digital industries |
Type Of Technology | Software |
Year Produced | 2012 |
Impact | A range of successful consultancies generating income of approx. £150000 |
Title | Kurv |
Description | I created a new musical interface as part of a partnership with Kurv Music. |
Type Of Technology | Physical Model/Kit |
Year Produced | 2015 |
Impact | This has led to the creation of a new music technology company, funded by a successful kickstarter campaign |
URL | https://www.kickstarter.com/projects/kurv/kurv-guitar |
Title | Maximilian |
Description | DSP library for interactive games, art and information retrieval |
Type Of Technology | Software |
Year Produced | 2009 |
Impact | the software is used by a wide range of artists and professionals as part of the openframeworks project |
URL | http://maximilian.strangeloop.co.uk |
Title | Oramics Simulator |
Description | An interactive Oramics machine simulator for the science museum's "Oramics" exhibition |
Type Of Technology | Software |
Year Produced | 2010 |
Impact | Has been described by the Science Museum as their "most popular interactive'. |
Title | Spider Sonics |
Description | Spider Sonics is an extension of LumiSonic that has been created as part of the Sound Image and Brain project in collaboration with a range of schools and colleges across the UK |
Type Of Technology | Software |
Year Produced | 2014 |
Open Source License? | Yes |
Impact | As yet the software has only been tested across three schools but is looking great! |
Title | The Dean Rodney Singers |
Description | iPhone and iPad app as part of the cultural olympiad, co-designed by disabled musicians. |
Type Of Technology | Software |
Year Produced | 2012 |
Open Source License? | Yes |
Impact | iPhone and iPad application co-designed by and for disabled musicians, released on the app store for free. |
Company Name | Kurv |
Description | Kurv develops a string-free digital guitar, two handheld controllers that allow the user to simulate playing guitar with no previous experience. It also develops a package that will allow users to design and create their own virtual instrument. |
Year Established | 2014 |
Impact | This company has received significant investment from third-party investors and successful crowdfunding campaigns. |
Website | http://www.kurv.io |
Description | AHRC Public Engagement Event |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Public/other audiences |
Results and Impact | I participated in the 2014 AHRC Creative Economy Showcase, with collaborating partners Heart n Soul and Roll7. Roll7 presented their game which was developed during the collaboration, which has since gone on to win a BAFTA |
Year(s) Of Engagement Activity | 2014 |
Description | Sky News Interview |
Form Of Engagement Activity | A press release, press conference or response to a media enquiry/interview |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Media (as a channel to the public) |
Results and Impact | I was interviewed by Sky news 'Swipe' as part of a special programme on new music technology. |
Year(s) Of Engagement Activity | 2016 |