Brain-Computer Interface for Monitoring and Inducing Affective States

Lead Research Organisation: University of Reading
Department Name: Sch of Biological Sciences

Abstract

Brain Computer Interfaces (BCI) allow for modification of subject's environment or control of external devices by the power of thought alone. They achieve this by analysis of small electrical potentials generated by the subject's brain while its owner is thinking. Emotions involve specific mental states hence also involve particular patterns of brains electrical activity. The proposed research will build innovative intelligent BCI systems that can monitor our emotions, and modify them automatically and adaptively via controlled computer music generation system. Creation of such systems would advance our understanding of fundamental relationships between the subjective emotions, corresponding brain states and characteristics of music that can induce very vivid and powerful emotions in humans. Such systems can be used for treatment of emotional/mood disorders such as depression so are of direct benefit to society and NHS. In addition, they are of interest for healthy subjects as means of relaxation or perhaps by enhancement of gaming experience. Thus the proposed project could also lead to interesting developments in the entertainment industries such as the gaming industry.

Planned Impact

This research will contribute towards the enhancement of the quality of life, health in the society and the economic competitiveness of the entertainment industry in the UK, including the gaming industries and music. Its interdisciplinary nature will also benefit several academic stakeholders as outlined in the Academic Beneficiaries section.
BCI technology to monitor and induce affective states will impact on the health sector by providing a valuable tool in preventive and therapeutic programmes for health problems such as stress, depression, anxiety and other conditions related to affective states. Stress is an endemic problem in our society and economy. According to a study by the mental health charity Mind (http://www.mind.org.uk/), every year UK businesses lose £26 billion and 70 million working days because of stress. Excessive stress leads to anxiety and more severe cases of depression. Stress may cause suppression of the immune system, insomnia, fatigue, and affect adversely productivity. Persistent cases may lead to social withdrawal and family dysfunction. The project impact will be enhanced by liaising with the health sector in order to raise their awareness on this projects goals and prepare the ground for taking our systems out of the lab into the real world of special needs as soon as we have a working prototype.
Activities that will enhance impact on creative industry include organisation of the Workshop during Peninsula Arts Contemporary Music Festival in Plymouth which will enable us to spread awareness about the technology developed in the music industry.
The project will positively impact public understanding of science via press exposure campaign, use of internet media (dedicated WWW page, youtube etc) for dissemination of the information about the project as well as by using the exploitation channels available at both universities.
Lastly, in additional to the usual channels of academic dissemination the impact on academia will be enhanced bythe organisation of the International Workshop at Reading as well as by providing a an excellent training for highly qualified interdisciplinary research staff.
 
Description A) Developed a systematic study to establish which musical parameters (characteristics) can be associated with affective states (emotions and feelings) that might be detectable in EEG patterns, as reported in
a. Daly, I., Malik, A., Hwang, F., Roesch, E., Weaver, J., Kirke, A., Williams, D., Miranda, E. R., Nasuto, S. J. (2014, June). Neural Correlates of Emotional Responses to Music; An EEG Study. Neuroscience Letters, Volume 573, Pages 52-57, ISSN 0304-3940, Elsevier.
B) Investigated music tempo as an alternative feedback mechanism in brain computer interface control and in neurofeedback, as reported in a number of project outputs:
a. Daly, I., Malik, A., Hwang, F., Roesch, E., Weaver, J., Kirke, A., Williams, D., Miranda, E. R., Nasuto, S. J (2014). Investigating music tempo as a feedback mechanism for closed-loop BCI control Brain-Computer Interfaces Brain Computer Interfaces, Volume 1, Issue 3, Taylor & Francis UK.
b. Daly, I., Williams, D., Hwang, F., Kirke, A., Malik, A., Roesch, E., Weaver, J., Miranda, E. R., Nasuto, S. J. (2014, September). Brain-computer music interfacing for continuous control of musical tempo. 6th International Brain-Computer Interface Conference 2014, Graz, Austria.
c. Daly, I., Hallowell, J., Hwang, F., Kirke, A., Malik, A., Roesch, E., Weaver, J., Williams, D., Miranda, E. R., Nasuto, S. J. (2014, August). Changes in music tempo entrain movement related brain activity. 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC'14), Chicago, Illinois, USA.
C) Proposed a method for identification of neural correlates of continuous variables, as reported in
a. Daly, I., Hwang, F., Kirke, A., Malik, A., Weaver, J., Williams, D., Miranda, E. R., Nasuto, S. J (2014). Automated identification of neural correlates of continuous variables Journal of Neuroscience Methods, 2014
D) Demonstrated that emotions evoked by music can be predicted from brain responses and acoustic features. Details of this can be found in
a. Daly, Ian; Williams, Duncan; Hallowell, James; Hwang, Faustina; Kirke, Alexis; Malik, Asad; Weaver, James; Miranda, Eduardo; Nasuto, Slawomir J. (2015) Music-induced emotions can be predicted from a combination of brain activity and acoustic features. Brain Cogn. 2015 Dec;101:1-11. doi: 10.1016/j.bandc.2015.08.003. E) Developed generative music algorithms aimed at inducing specific affective states F) Developed machine learning techniques able to learn complex EEG patterns and make decisions to steer generative music algorithms in real-time.
G) Implemented a number of BCI prototypes for musical control to test complementary methods for controlling music. These investigations are detailed in
a. I Daly, D Williams, A Malik, J Weaver, A Kirke, F Hwang, E Miranda, SJ Nasuto, "Personalised, Multi-modal, Affective State Detection for Hybrid Brain-Computer Music Interfacing," in IEEE Transactions on Affective Computing, vol. PP, no. 99, pp. 1-1, 2018. doi: 10.1109/TAFFC.2018.2801811
b. Daly, I., Williams, D., Hwang, F., Kirke, A., Malik, A., Roesch, E., Weaver, J., Miranda, E. R., Nasuto, S. J., (2015). Towards Human-Computer Music Interaction: Evaluation of an Affectively-Driven Music Generator Via Galvanic Skin Response Measures. IEEE 7th Computer Science & Electronic Engineering Conference (CEEC'15). Essex, UK, 24-25 September 2015.
H) Developed machine learning techniques able to learn complex EEG patterns and make decisions to steer generative music algorithms in real-time. The implementation and its evaluation have been reported in a number of outputs: a. I Daly, D Williams, A Malik, J Weaver, A Kirke, F Hwang, E Miranda, SJ Nasuto, "Personalised, Multi-modal, Affective State Detection for Hybrid Brain-Computer Music Interfacing," in IEEE Transactions on Affective Computing, vol. PP, no. 99, pp. 1-1, 2018. doi: 10.1109/TAFFC.2018.2801811
b. Affective Brain-Computer Music Interfacing. Daly, I., Williams, D., Hwang, F., Kirke, A., Malik, A., Weaver, J., Miranda, E. R., Nasuto, S. J., (2016) Journal of Neural Engineering, 13:4, July 2016. http://dx.doi.org/10.1088/1741-2560/13/4/046022
c. An affective brain-computer music interface Daly, I., Williams, D., Kirke, A., Weaver, J., Malik, A., Hwang, F., Eaton, J., Miranda E.R., Nasuto, S.J., International Brain-Computer Interface (BCI) Meeting, California, USA, May 30-June 3 2016.
d. Daly, I., Williams, D., Hwang, F., Kirke, A., Malik, A., Roesch, E., Weaver, J., Miranda, E. R., Nasuto, S. J., (2015). Identifying music-induced emotions from EEG for use in brain-computer music interfacing. 4th Workshop on Affective Brain-Computer Interfaces at the 6th International Conference on Affective Computing and Intelligent Interaction (ACII2015). Xi'an, China, 21-25 September 2015.
I) Developed methods to analyse interaction patterns in the EEG signal and shown their modulation by music tempo
a. N Nicolaou, A Malik, I Daly, J Weaver, F Hwang, A Kirke, EB Roesch, D Williams, ER Miranda, SJ Nasuto, Directed motor-auditory EEG connectivity is modulated by music tempo, Frontiers in human neuroscience 11, 502, 2017.
J) Gained a better understanding of how music can change affective states and what are the neural correlates of such affective responses.
a. Ian Daly, Duncan Williams, Faustina Hwang, Alexis Kirke, Eduardo Miranda, Slawomir J. Nasuto, Neural circuits underlying emotional responses to music: A joint EEG-fMRI study (in submission)
K) Run a pilot feasibility study of the developed affective BCI system testing its usability with patients with neurological disorders
a. Affective Brain Computer Music Interfacing: A Case Study Of Use By An Individual With Huntington's Disease. Daly, I., Williams, D., Kirke, A., Weaver, J., Malik, A., Hwang, F., Miranda E.R., Nasuto, S.J., in 7th Graz BCI Conference 2017, Austria.18-22 September 2017.
Exploitation Route We believe that (a) the methods that we developed for EEG analysis for detecting effective states, (b) the algorithms for generating music aimed at inducing specific effective states, (c) the methods we develop for regulating EEG patterns using music and (d) the Brain-Computer Interfacing (BCI) prototypes and proof-of-concepts that we implemented and tested, have paved the ground for the development of practical musical BCI systems for regulating affective states (e.g., to induce specific emotions/feelings). Thus, these results pave the way for more systematic incorporation of music as a viable feedback channel in Brain Computer Interfaces and open up the possibility of using such technology for a wide variety of therapeutic applications, e.g. those targeting affective disorders.

Also, the method for identification of neural correlates of continuous variables might be of interests to BCI practitioners designing new BCI systems but also for researchers trying to characterise the neural processes associated with some cognitive functions.
Sectors Creative Economy,Digital/Communication/Information Technologies (including Software),Healthcare,Leisure Activities, including Sports, Recreation and Tourism

URL http://cmr.soc.plymouth.ac.uk/bcmi/
 
Description The project generated quite a lot of interest in the general public via outreach activities, for example the Guardian feature involving a pop musician Tiny Tempah. Several other person from music industry also expressed interests in the project outcomes and a possibility of some form of future collaboration. Also, a PR company has recently contacted Prof Nasuto about a possibility of conducting a short study on affective consequences of live concert music experience. Following a pilot evaluation of the developed prototype affective music BCI system wiht the Huntington Disease patient, we have been exploring the potential deployment of systems build around affective generation of music in therapeutic context and we continue our interactions with clinical researchers to pursue funding opportunities for further testing and expansion of the system towards therapy applications in dementia. Moreover, Prof Nasuto developed a collaboration with an anaestesiologist from Westimnster and Chelsea Hospital and is investigating a possibility of developing of closed loop music system within the peri-operative context. One potential outcome of such an application would be a reduction of anxiolytic medications taken up by the patients patients and decreased incidence of mental deficits which have been recognised as one of the undesirable side effectis of the peri-operative care.
Impact Types Cultural

 
Description BBC News visited the ICCMR and interviewed Professor Miranda about the application of Music Neurotechnology in his piece Activating Memory 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Public/other audiences
Results and Impact the interview was aired on national radio.

the information about our research reached wider audience.
Year(s) Of Engagement Activity 2011,2014
 
Description Dr E. Roesch and Prof SJ Nasuto interviewed for a Feature programme in French TV channel Encyclo devoted to BCI technology 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact Raised the awarness in assistive technologies and Brain COmputer Interfaces

The programme has not been aired yet.
Year(s) Of Engagement Activity 2014
 
Description Neural Engineering Transformative Technologies (NETT) summer school 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Postgraduate students
Results and Impact Talk raised awareness about the research at the crossroads of music and neural engineering.

Increased awareness on state of the art research on music and neural engineering.
Year(s) Of Engagement Activity 2013
 
Description Professor Miranda Symphony piece of Minds Listening which was premiered at the Peninsual Arts Contemporary Music Festival in Plymouth 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? Yes
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact audience could experience music created with the use of the technology developed partly through the project.

Stimulated public understanding and appreciation of the research.
Year(s) Of Engagement Activity 2013
 
Description The Engineer previewed the project, including interviews with principal investigators Eduardo Miranda and Slawomir Nasuto. 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact Further commentary from record producer Steve Levine (the Beach Boys, Culture Club) on some of the real-world applications for the results were discussed.

Raised awareness on the research in general public.
Year(s) Of Engagement Activity 2011
 
Description The Guardian feature 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact The Guardian made a feature on the music and and tis effect on emtion and the brain. The feature was centred around a visit by a pop musician, Tiny Tempah, to the University of Reading where we discussed the project and performed a scan of his brain while he was listening to various musical pieces.
Year(s) Of Engagement Activity 2016
URL https://www.theguardian.com/lifeandstyle/2016/sep/12/mind-blowing-music-tinie-tempahs-brain-scan
 
Description interview for spotify 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact A company preparing promotional material for Spotify for CES2017, a global consumer electronics and consumer technology tradeshow, performed interview with me on 'power of audio'.
Year(s) Of Engagement Activity 2016