Towards Detecting Pilots' Work Overload From their Brain Signals - Next Generation Cockpit Information System

High pressured and complex information environments such as those experienced by fighter pilots place high cognitive loads (termed "cognitive overload") on the pilots. This is due to the vast amount of potentially critical information that different instruments and sensors provide in various modalities. Pilots need to focus on important information related to suspicious events and targets, while still maintaining overall comprehensive situational awareness. A delay in observing, comprehending and in turn, reacting to critical information could mean the difference between life and death. This means that pilots have to utilise and engage with the right instrument and sensor at the right time and use that information to make the best possible decision all within a very short space of time. Research has shown that cognitive overload can happen when users are engaged in multiple tasks simultaneously and, in particular, if the tasks have different modality in nature. For example, pilots have to interact and respond to the cockpit information system (visual modality) while they are communicating with their wingmen or command control (auditory modality). Also, research has shown that users can experience cognitive overload for one modality (e.g. visual) while still able to receive information from another type of modality (e.g. auditory) and process it. This is particularly important if, for example, a pilot is experiencing a visual cognitive overload, sending mission-critical information via such modality would be ineffective and highly likely missed by the pilots. However, sending the same information using another modality (e.g. auditory) would allow the pilots to receive and process the information at the intended time. Having this knowledge about pilots will increase the success of delivering critical information by cockpit information systems which could, in turn, save pilots' life. The overall goal of this research is to make a fundamental first step in establishing that pilots' cognitive overload can be detected based on a series of brain activations and reacted upon them by cockpit information systems. Early research in neuropsychology has indicated that it is possible to monitoring cognitive load. However, this research is still in its infancy. This project will push the boundaries of science by establishing that cognitive overload can be detected based on a series of brain activations. We also aim to go one step further and detect the modality of cognitive overload, i.e. either it is visual, auditory or kinaesthetic, in real-time, from the brain signals. Finally, we aim to investigate if the brain activates that have been identified are common across users, or they are unique to each individual. During the course of the project, neuroscience monitoring and modelling techniques will be employed to localise brain regions that get activated when participants are experiencing cognitive overload. Artificial Intelligence models will then be devised to learn and detect from the brain signals (of those localised regions) whether or not a user is experiencing a cognitive overload and what modality of cognitive overload would that be at a given time.