Building a Boolean framework of multisensory decision making

The development of multiple senses is a major evolutionary success story. A multisensory organism generally benefits as it can discover complex relations between biologically relevant events and so develop a broad understanding of its environment. Moreover, by combining redundant signals on one event (e.g., vision and audition can provide redundant location signals), the organism can act faster and more accurately, which in many situations can be life-saving. Analysing signals from the different senses to enable multisensory benefits is consequently one of the most important cognitive brain functions and much research has been dedicated to unravel specifically the computational principles involved in the processing of redundant signals. It is however surprising that there is currently no unified framework to address the general case. This proposal is consequently about asking what are the computational principles that allow an organism not only to benefit from redundancy but to generally understand any relation between sensory signals to control its action?

To develop a general framework, the proposed research builds on powerful models of perceptual decision making, which have been developed to understand behavioural responses when participants are for example asked to perform a simple motor act in response to a sensory signal. The task may appear abstract but applies to any real-life situation, in which a sensory signal requires action (e.g., consider a car driver, who gives way when hearing the wailing sirens of an ambulance). To extend the successful uni-sensory models to multisensory decisions, I have recently demonstrated that unisensory decisions in audition and vision can be flexibly coupled by logic decision gates that perform AND/OR operations. Here, I propose that these findings can be taken much further as they may in fact point to a much broader computational principle: Boolean algebra. Being rooted in the work of George Boole, the computational power of Boolean algebra is fundamental to all modern digital electronics. Its basic operators are the conjunction (AND), disjunction (OR), and negation (NOT) that take as in- and output the values true and false. The striking question that I ask here is whether Boolean algebra can also help to understand the cognitive architecture that extends unisensory to multisensory decisions? To answer this question, three critical gaps need to be closed:

(1) We need to know whether the proposed logic gates can be generalized. For example, can tactile signals be combined using the same AND/OR decision gates as found for vision and audition? If yes, this would allow unisensory decisions to be conceptualized as truth values of Boolean algebra.

(2) We need to know whether the proposed logic gates can be extended. For example, can the AND/OR decision gates be put together like Lego bricks? This would show that, as in computers, complex operations could be performed by basic operations.

(3) We need to know whether the proposed logic gates form a complete set, which is the minimal set of basic functions that are needed to build any complex function. Following Boolean algebra, a third basic function is the negation, which is however underexplored and needs to be investigated.

To develop and test a general computational neuroscience framework of multisensory processing, the proposed research will address each gap on two levels. First, human behavioural experiments will describe the timing of complex multisensory decisions that require combination of distinct sensory signals from vision, audition, and touch. Second, the behavioural experiments will be guided by computational models to understand the underlying cognitive functions. At the end, this will enable us to also understand, and crucially predict, the decisions made by a car driver, who does not start driving when seeing the traffic light turning green and when hearing the wailing sirens of an ambulance.

To interact with the environment, animals including humans constantly use signals from the different sensory modalities like vision, audition, and touch. Much research has been dedicated to understand the underlying processes and computational principles especially when redundant signals are combined (e.g., when vision and audition can provide redundant location information). However, there is currently no general computational framework to understand how in principle any (and not only redundant) signals from the different senses are used to control actions. To develop such a general computational framework of multisensory processing, we here conceptually extend the perceptual decision making framework, which has proven to be extremely powerful with unisensory decisions but which has been used far less for considering what happens when more than one sense is used, to multisensory situations. Based on our previous findings that unisensory decisions can be coupled by logic decision gates that perform AND/OR operations, we propose that multisensory processing can be more generally understood using Boolean algebra, which is the ground-breaking subarea of mathematics fundamental to all our modern digital electronics. Within this framework, we conceptualize the categorical decisions made with uni-sensory signals as the truth values of Boolean algebra, which consequently can be coupled by three basic logic operators, which are the conjunction (AND), disjunction (OR), and negation (NOT), or more complex composed logic operators. In this project, we will follow an experimental approach to test the timing of human decision making behaviour combined with computational modelling to test and develop this general framework of multisensory decision making, which we argue has the potential to reveal core cognitive functions of reasoning.

My proposed project is for fundamental research in cognitive and computational neuroscience with no direct application to UK economy. However, I have identified potential stakeholders that will benefit from the knowledge generated from the proposed research:
Medical professions involved in symptoms and treatments of mental disorders
The proposed research will be relevant for medical professionals interested in cognitive and other categories of mental health disorders (e.g., developmental dyslexia and autism). Although potential benefits lie certainly downstream of this application, distinct actions can be taken to maximise the impact of my research, which investigates core cognitive functions in healthy humans. By developing a general computational framework, we can provide detailed understanding of normal cognitive functioning. In the long run, this knowledge has a huge potential to be applied in clinical research to identify specific deficits in cognitive functioning in patient populations. To guarantee efficient application, I will provide the largest possible support to clinical researchers, as I have done in the past. Moreover, we aim to publish, or make available on my Lab website, detailed analysis toolboxes which are ready and easy to use for clinical researchers. In addition, we will take active steps to promote potential application in relevant interest groups. For example, I have identified the Scottish Autism Research Group (SARG), which is a platform for sharing, debating and building upon current research on autism with the goal to encourage inter-disciplinary collaboration. I joined the SARG and aim to contribute to this interactive platform by giving public talks.
The general public
My proposal, in its essence, connects computational principles to human cognitive functions. As such, it relates to topics including the development of modern computer technology, the brain-computer metaphor, and at the end of the day to the question of who we are. These topics frequently elicit immense public interest, which presents itself for example in recent highlights from the cinemas like the drama The Imitation Game (2014, directed by Morten Tyldum), which is based on the biography of mathematician and pioneering computer scientist Alan Turing, or the British science fiction thriller Ex Machina (2015, directed by Alex Garland), which evaluates human qualities in a fictional breakthrough experiment on artificial intelligence. To use the film as a vehicle to engage with the general public in scientific discussions, I contacted the Dundee Contemporary Arts (DCA) to discuss possible formats of a science film festival with the focus on computational and cognitive neuroscience. The DCA is a vibrant social and cultural hub with the mission to foster community and education. We already short-list potential movies that each will be accompanied by a scientific talk leading to discussion with the audience. The festival will be introduced by a public lecture connecting my research to the specific scientific topics addressed by the selected films. We have identified the Dundee Science Festival as the target to start the film festival.
Individuals involved in the project
For the individuals involved in the project, the research will provide training and expertise in important research skills including project planning and management, team work, as well as communication to both scientific and lay audiences. A special focus will be on the training of the postdoctoral research assistant, who will have challenging stages ahead of him or her to build a career in academia. To promote career development, the project aims to provide the largest possible support, for example, by also providing first-class professional training with strong networking potential (as available in scientific summer schools). By committing to public engagement, I see the potential for intellectual growth for all individuals involved in the project.