TASCC: Driver-Cognition-Oriented Optimal Control Authority Shifting for Adaptive Automated Driving (CogShift)
Lead Research Organisation:
Cranfield University
Department Name: Sch of Aerospace, Transport & Manufact
Abstract
The emerging development of automated driving demands a mutual understanding and a smooth coordination between human driver and vehicle controller, so as to avoid conflict and mismatch in demands, and instead achieve desirable driving performance, smooth and swift transitions which enhance driving safety during complex operating scenarios. However, such driver-vehicle collaboration during automated driving will impact on the driver's attention and cognition and it is important to consider these effects in order to prevent any negative impact on driving. This project aims to achieve a safe engagement and smooth and swift control-authority shift between the driver and the vehicle controller during adaptive automated driving. To this aim, we will first conduct a comprehensive study of driver attention and cognitive control characteristics when interacting with the vehicle controller. An optimal control authority shifting system which considers driver cognition will then be systematically developed and validated. This cross-disciplinary research challenge will be addressed using a unique combination of researchers from engineering, cognitive neuroscience and human factors. The research will not only contribute to the cutting-edge technology innovations in automated driving, but will also result in a major advance in the science of human attention and cognitive control when interacting with automation.
Planned Impact
This research aims to make a major scientific advance in human attention and cognitive control when interacting with automation, and cutting-edge technology innovations in automated driving, using a unique combination of internationally known experts from engineering, cognitive neuroscience and human factors. The research along with its outputs will benefit a wide range of groups, including:
- Academic researchers worldwide in disciplines such as engineering, cognitive neuroscience, psychology and human factors. These will not only benefit from the research itself, but also gain insights from such a unique combination of cross-disciplinary expertise to tackle complex research challenges that are beyond a single discipline.
- All stakeholders related to the automotive sector will significantly benefit from this research by taking advantage of its research methodology and framework, as well as the outcomes and technological innovations. This will contribute to the emerging development of automated cars, in particular future-generations of JLR's intelligent and automated cars.
- Government and policy-makers will benefit from this research, when developing regulations, recommendations, and training related to safe automated cars and driving.
The research outcomes will also contribute to the enhancement of driving safety by generating a smooth control-authority shifting collaboration between the driver and the vehicle controller. The research will further help to promote public awareness of automated cars and their benefits.
- Academic researchers worldwide in disciplines such as engineering, cognitive neuroscience, psychology and human factors. These will not only benefit from the research itself, but also gain insights from such a unique combination of cross-disciplinary expertise to tackle complex research challenges that are beyond a single discipline.
- All stakeholders related to the automotive sector will significantly benefit from this research by taking advantage of its research methodology and framework, as well as the outcomes and technological innovations. This will contribute to the emerging development of automated cars, in particular future-generations of JLR's intelligent and automated cars.
- Government and policy-makers will benefit from this research, when developing regulations, recommendations, and training related to safe automated cars and driving.
The research outcomes will also contribute to the enhancement of driving safety by generating a smooth control-authority shifting collaboration between the driver and the vehicle controller. The research will further help to promote public awareness of automated cars and their benefits.
Publications
Zhao Y
(2017)
An Orientation Sensor-Based Head Tracking System for Driver Behaviour Monitoring.
in Sensors (Basel, Switzerland)
Yang L
(2022)
Learning Spatio-Temporal Representations With a Dual-Stream 3-D Residual Network for Nondriving Activity Recognition
in IEEE Transactions on Industrial Electronics
Yang L
(2021)
A Refined Non-Driving Activity Classification Using a Two-Stream Convolutional Neural Network
in IEEE Sensors Journal
Yang L
(2021)
Recognition of visual-related non-driving activities using a dual-camera monitoring system
in Pattern Recognition
Yang L
(2020)
A Dual-Cameras-Based Driver Gaze Mapping System With an Application on Non-Driving Activities Monitoring
in IEEE Transactions on Intelligent Transportation Systems
Xing Y
(2018)
Driver workload estimation using a novel hybrid method of error reduction ratio causality and support vector machine
in Measurement
Nagle Fintan S.
(2019)
Modelling Visual Complexity of Real-World Scenes
in PERCEPTION
Nagle F
(2020)
Predicting human complexity perception of real-world scenes.
in Royal Society open science
| Description | Within the scope of the project activities, a number of achievements have been obtained so far, which are concisely summarised below. i). An online survey on what types of non-driving tasks (NDT) a human driver is likely to be performing when travelling in a Level 3 automated vehicle has been conducted and documented in an internal report. Four activities were identified as the most important NDTs during a 60-minute journey in such a case, namely internet browsing, reading, news and emailing. Based on these findings, experiments have been designed and carried out to understand human driver attentional load in internet picture browsing during the autonomous driving mode. The experiments collected diverse images representative of internet browsing, obtained reliable crowd sourced ratings of visual attentional load, and helped to construct and fit a computational model using Convolutional Neural Network to predict such attentional demand. Comparisons against two prominent salience models, MLnet and SalGAN, indicated the novelty of the proposed work. ii). For the purpose of measuring and monitoring task load using EEG, preliminary research was conducted. Behavioural work established a data set of meaningful scene images that have been ranked on their level of visual complexity. This data set will be serve as a basis for the future study. iii). A prototype system for online driver NDT detection has been developed and installed/evaluated in a test car. The prototype focused on measuring driver head and limb motions while travelling in a car. This system was designed in a hybrid fashion with the integration of one camera and two motion sensors. Preliminary experimental tests showed that the fusion of two measurements was found to improve the reliability of head tracking accuracy. The experimental results also indicated that different NDTs led to different head movement patterns, which can be applied as an important feature to determine the type of NDTs. This research demonstrates the potential of using such a system as a basis for the development of an online NDT and driver attention detection system. Work is on-going (outside the scope of this study) to improve the computational performance and make the process real-time capable. If achieved this can make the process suitable for commercial application. iv). To understand driver's take-over behaviours so as to prepare for the design of CogShift take-over system, analysis of Autopilot disengagements (DE) occurred during automated vehicle testing was performed by reviewing seven public DE reports. Reasons of DEs were reviewed such as hardware issues, software failures, weather conditions, road surface conditions, emergencies and precautionary intervention, among which the software issues and limitations were the most common. From this research, take-over mechanisms have also been preliminarily investigated and it has been found that the average take-over time was less than one second. Based on the results, recommendations were provided to auto OEMs and governmental organisations. |
| Exploitation Route | The key findings might benefit and be taken forward by researchers in cognitive neuroscience, automotive engineering, information and communications technology, and human factors. The online survey results (a paper planned to be submitted) would provide a starting point for researchers to understand human activities when interacting with automated driving. The novel way of computationally modelling visual attentional load relating to image complexity together with the preliminary results for EEG markers would offer a base for the study of cognitive load engaging in a visual-related non-driving task. The prototype online detection system and relate findings (a paper planned to be submitted) would provide a valuable tool in developing driver state monitoring systems based on head motions. The review of disengagement activities during automated driving and findings (a journal paper under review) would not only benefit researchers who develop automated vehicle technologies, but also have potential commercial impact and would provide valuable information for automotive R&D and also policy makers. The postdoctoral researchers and PhD students within the project have been steadily developing research as well as transferable skills, which will benefit their career life. The project has generated three patent applications currently being processed by the commercial sponsor (Jaguar Land Rover), and a number of the occupant measurement tools have a direct commercial application within JLR and are now being used by JLR research teams in the development of future vehicles. An ESPRC IAA award has been won and is currently being executed to further develop the occupant measurement aspects of the project. |
| Sectors | Aerospace, Defence and Marine,Communities and Social Services/Policy,Digital/Communication/Information Technologies (including Software),Education,Electronics,Government, Democracy and Justice,Manufacturing, including Industrial Biotechology,Transport |
| Description | The findings are currently being assessed for the potential for three patent applications, which are currently being drafted by the sponsor. |
| First Year Of Impact | 2018 |
| Sector | Digital/Communication/Information Technologies (including Software),Education,Manufacturing, including Industrial Biotechology,Transport |
| Impact Types | Societal,Economic,Policy & public services |
| Description | Educational research opportunities for postgraduates studying for MSc and PhD degrees |
| Geographic Reach | Multiple continents/international |
| Policy Influence Type | Influenced training of practitioners or researchers |
| Impact | Students from a range of geographic backgrounds have been offered MSc and/or PhD research opportunities based on CogShift and similar research possibilities that it has opened up. |
| Description | New MSc in Connected and Autonomous Vehicle Engineering (Automotive) launched |
| Geographic Reach | Multiple continents/international |
| Policy Influence Type | Influenced training of practitioners or researchers |
| Impact | In 2020, Cranfield University launched a new MSc in Connected and Autonomous Vehicle Engineering (Automotive) which draws on the know-how developed during the CogShift project and other similar activities. The MSc covers a variety of areas relevant to practitioners including human-computer interaction. In particular, the expertese developed running trials involving automated vehicles is being exploited. The course has been attended by an international student body; some of the students are likely to remain in the UK, but others are likely to work elsewhere internationally. |
| URL | https://www.cranfield.ac.uk/courses/taught/connected-and-autonomous-vehicle-engineering-automotive |
| Description | EPSRC IAA - Development of driver behaviour tracking technology for semi-autonomous and human-driving vehicles |
| Amount | £37,854 (GBP) |
| Funding ID | EP/R511511/1 |
| Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
| Sector | Public |
| Country | United Kingdom |
| Start | 04/2020 |
| End | 03/2022 |
| Title | Creation of dual drive test vehicle for auntomoy hand over studies. |
| Description | The tool consists of a research vehicle capable of simulating a humanistic level of driving autonomy which can then be used to assess human responses to autonomous and manual driving and the switch between the two modes. The vehicle is based on a Land rover Discovery and has a normal driving position in the front and a second driving position in the rear. It is also fitted with a bespoke control and DAQ system, with occupant monitoring via internal cameras. The vehicle is to be used in conjunction with the Cranfield MUEAVI (Multi user environment for autonomous vehicle innovation) test track, which was developed specifically for this purpose but not as part of this program of work. |
| Type Of Material | Improvements to research infrastructure |
| Year Produced | 2018 |
| Provided To Others? | No |
| Impact | This tool enables the validation of existing theories with respect to hand over between autonomous driving and manual driving. The research vehicle and experimental methodology will now be used to develop haptic feedback systems to aid the takeover sequence of events in a safe manner. |
| Title | Head tracking system for autonomous vehicle research |
| Description | Two head tracking systems have been developed for the measurement of occupant head movement within a moving vehicle. One system is based on an orientation sensor and one on a camera based system. Both systems are in use by the research project to measure aspects of occupant movement as part of the wider research project aims associated with driver attention levels during non driving tasks. The data these systems provide is then analysed using a movement correlation approach to characterise driver attention level. |
| Type Of Material | Physiological assessment or outcome measure |
| Year Produced | 2017 |
| Provided To Others? | No |
| Impact | The tool is in use currently as part of the research program, and hence only has local impact currently, however it is expected to be used for further research in this area over the coming years. |
| Title | Predicting human complexity perception of real-world scenes |
| Description | Perceptual load is a well-established determinant of attentional engagement in a task. So far, perceptual load has typically been manipulated by increasing either the number of task-relevant items or the perceptual processing demand (e.g. conjunction vs. feature tasks). The tasks used often involved rather simple visual displays (e.g. letters or single objects). How can perceptual load be operationalised for richer, real-world images? A promising proxy is the visual complexity of an image. However, current predictive models for visual complexity have limited applicability to diverse real-world images. Here we modelled visual complexity using a deep convolutional neural network trained to learn perceived ratings of visual complexity. We presented 53 observers with 4000 images from the PASCAL VOC dataset, obtaining 75,020 2AFC paired comparisons across observers. Image visual complexity scores were obtained using the TrueSkill algorithm. A CNN with weights pre-trained on an object recognition task predicted complexity ratings with r=0.83. In contrast, feature-based models as used in the literature, working on image statistics such as entropy, edge density and JPEG compression ratio, only achieved r = 0.70. Thus, our model offers a promising method to quantify the perceptual load of real-world scenes through visual complexity. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2019 |
| Provided To Others? | Yes |
| URL | http://datadryad.org/stash/dataset/doi:10.5061/dryad.3fs556j |
| Company Name | MINDVISIONLABS LIMITED |
| Description | Mind Vision Labs conducts complimentary research in machine learning and brain sciences to create a more human-like artificial intelligence for autonomous systems. |
| Year Established | 2017 |
| Impact | Mind Vision Labs has got currently 4 FT postdoctoral scientists and a fifth is being under recruitment. So far it has filed for two patents and published one paper. |
| Website | http://mindvisionlabs.com/ |
| Description | Interview for local radio (BBC Three Counties) |
| Form Of Engagement Activity | A broadcast e.g. TV/radio/film/podcast (other than news/press) |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Media (as a channel to the public) |
| Results and Impact | DA gave an interview to BBC Three Counties Radio on 20 August 2020 talking about Cranfield's work on driverless cars. This was a short, five minute interview aimed at engaging the wider public in some of the work we do. |
| Year(s) Of Engagement Activity | 2020 |
| Description | Jaguar Land Rover TASCC - Share Fair exhibition event |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Industry/Business |
| Results and Impact | The event comprised of presentations and demonstrations of the research to an audience within Jaguar Land Rover. Held at the Heritage Motor Museum at Gaydon, the day long event attracted a large number of Jaguar Land Rover employees, and had the aim of disseminating the research results to date to the wider business. |
| Year(s) Of Engagement Activity | 2018 |
| Description | Jaguar Land Rover TASCC - Share Fair exhibition event 2019 |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Industry/Business |
| Results and Impact | The event comprised of presentations and demonstrations of the research to an audience within Jaguar Land Rover. Held at the Heritage Motor Museum at Gaydon, the day long event attracted a large number of Jaguar Land Rover employees, and had the aim of disseminating the research results to date to the wider business. |
| Year(s) Of Engagement Activity | 2019 |
| Description | Prof Lavie interviewed about limited capacity for the New Scientist |
| Form Of Engagement Activity | A press release, press conference or response to a media enquiry/interview |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Media (as a channel to the public) |
| Results and Impact | Prof Lavie's interviwed for an article centered on her recentl research titled "Struggling to multitask? Your brain might have hit full capacity" the work has been explicitly related to driver's safety |
| Year(s) Of Engagement Activity | 2018 |
| URL | https://institutions.newscientist.com/article/2178366-struggling-to-multitask-your-brain-might-have-... |
| Description | Prof Lavie interviewed for a feature article and cover story in the Geo magazine (the German equivalent of Nat. Geo) |
| Form Of Engagement Activity | A magazine, newsletter or online publication |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Media (as a channel to the public) |
| Results and Impact | Prof Lavie was interviewed for the German Geo about "Focus: Less distraction, more clarity" (feature article and cover story). This article should contribute to increased awareness of the importance of attention focus for driving safety among other daily activites and work tasks |
| Year(s) Of Engagement Activity | 2019 |
| URL | https://www.geo.de/magazine/geo-magazin/33886-geo-nr-02-2019-fokus-weniger-ablenkung-mehr-klarheit |
| Description | Prof Lavie joined the House of Lords consultancy pannel on green energy (chaired by Lord Howel) as a presenting member |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Policymakers/politicians |
| Results and Impact | Prof Lavie presented a speech on the impact of autonmous car techonogy on green eneregy |
| Year(s) Of Engagement Activity | 2017 |
| Description | Prof Lavie was interviewed on BBC News |
| Form Of Engagement Activity | A press release, press conference or response to a media enquiry/interview |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Media (as a channel to the public) |
| Results and Impact | On 13.08.2019 Prof Lavie was interviewed for a BBC News piece titled "Hands-free phone ban for drivers 'should be considered'. BBC News |
| Year(s) Of Engagement Activity | 2019 |
| URL | https://www.bbc.co.uk/news/uk-49320473 |