iSeat - Towards an intelligent driver seat for autonomous cars

Lead Research Organisation: University College London
Department Name: Civil Environmental and Geomatic Eng

Abstract

According to the most optimistic predictions, the first commercially available fully-autonomous cars are expected in 2040 offering the consumer a full end-to-end journey. These self-driving vehicles will be equipped with technology allowing autonomy Level 5 in which there is no interference required by the human. The concept of autonomy levels was first published by the international Society of Automotive Engineers in 2014. The report defines six levels of autonomy that automakers would need to achieve on their way to building the no-steering-wheel self-driving bubble pods of the future reaching from the fully-manual Level 0 to the fully-autonomous Level 5.

In the race towards the first commercially available fully-autonomous car, the majority of cars on UK roads will be equipped with technology that allows Level 3 or 4 autonomy over the next two decades. Drivers will be provided with increasingly sophisticated features such as lane-keep and steering assist. These semi-autonomous cars might be able to transport the driver autonomously on sections of a journey. However, the driver is required to take control occasionally between different levels of autonomy when required to complete an end-to-end journey. These transitions between autonomy levels cause safety concerns, as the driver might not be fully aware of the surrounding situation and the enabled autonomy features instantly.

This project proposes a new interface design for semi-autonomous cars called iSeat. This system is fundamentally different compared to current systems (such as Tesla's Autopilot or DistronicPlus by Mercedes) using visual or auditory indications which might be mentally overloading and distracting for the driver. iSeat is an intelligent driver seat acting as a co-pilot measuring the current mental and physical engagement of the driver and allowing safe, coordinated and timely transitions between different levels of autonomy. Of particular significance is the driver seat made of robotic structures serving the feedback purpose as well as providing monitoring capabilities through direct contact with the human during any level of autonomy: Tactile sensation can be fed back to the driver, the seat ergonomics and comfort can be changed and the robotic structures can measure the pressure distribution of the driver's weight. iSeat sensing information will be fused with multi-modal sensing data from electrical activity produced by skeletal muscles (Electromyography (EMG) signals) and in the driver's brain (Electroencephalography (EEG) signals), and input from vision cameras regarding the driver's posture and the point of gaze (i.e. where the driver is looking). This real-time knowledge will be classified through machine learning and affective interaction techniques in terms of the awareness state of the driver. Personalised feedback will be provided (i.e. tactile sensation, stiffness feedback, change of the driver seat ergonomics/comfort, visual/auditory feedback) to support the driver so that safe, timely, effective and intuitive transitions between different autonomy levels can be completed.

The iSeat system builds upon a complete re-think of the manner in which humans interact with autonomous cars. The smart combination of sensor systems, machine learning, affective computing, human factors, haptics and robotics will result in a bi-directional human-machine cooperation that is safe, intuitive, effective, and personalised.

Planned Impact

Who will benefit from the project?

This outcomes of this project will benefit (i) the general public, i.e. people who will travel in autonomous cars that cannot offer full end-to-end journey, (ii) the UK industry, i.e. carmakers, aiming at producing autonomous cars, and the technology industry, that supplies carmakers with systems that will be integrated into cars and (iii) academics in the area of human factors, robotics and transport engineering. Other major beneficiaries will be policy makers who will investigate ways to standardise procedures in autonomous cars to ensure safety.

How will they benefit from the project?

The general public - Autonomous vehicles will transform the transportation industry beyond the automotive and technology industries and affect delivery services and public transport even in remote areas and dispersed communities. An autonomous vehicle will certainly be, for the majority of people, the first "autonomous robot" that they are interacting with on a daily basis and that might make decisions through AI that can be life-threatening. To overcome any lack of trust and confidence in these autonomous robots, which might lead to any constraints of public acceptance, iSeat addresses human-machine cooperation risk factors and introduces intelligent sensing and intuitive (haptic) feedback technologies into the driving environment.

The UK industry - The annual economic benefits of autonomous cars is predicted to grow to £51 and £121 billion by 2030 and 2040 respectively. Reliability and safety concerns can be extremely damaging for carmakers. For instance, after Tesla's Autopilot crash in 2016, new tighter regulations were released on testing automation features. Tighter regulation could slow the development and introduction of new technology. In worse cases, new developments might be aborted due to high-risk commercialisation factors. Further, autonomous cars are transformative for the automotive industry at large - acting as a catalyst for new products and technology. The introduction of autonomous cars will revolutionise safety. iSeat aims at reducing safety risks involved in the process of taking back control from a car or vice versa, which will lead to direct economic impacts to various road industries such as the automotive, coach, trucking, ground shipping, as well as, other transportation sectors such as aviation, maritime and rail. This will help the UK to become world-leading in the development of technological products and to attract experts for human-machine cooperation of autonomous cars.

Academics - Due to the high-risk nature of the proposed work, this project will primarily produce new knowledge and understanding on how various feedback modalities can support drivers to gain the right level of engagement/vigilance, optimally complete procedural steps between different levels of autonomy, and efficiently get back into the control loop in various driving situations. The knowledge gained throughout this project will be beneficial for other application areas such as the integration of collaborative robots that work closely together with the human in industrial settings.

Publications

10 25 50
 
Description This project effectively started on 2 Jan 2019 after 2 rounds of recruitment (interview process). We developed a reconfigurable structure as a haptic feedback interface in a driver seat to guide drivers during transition between different levels of autonomy. We have made a review on existing situation awareness and detection systems for transition between autonomy level in (highly) automated vehicles. We have also collected and analysed data on situational awareness and we tested multiple models to establish a relationship between signals received from a human brain and situational awareness.
Exploitation Route We are currently exploring a number of robotic origami systems that can be used as haptic feedback actuators informing the driver of transitions between different levels of autonomy. This haptic feedback system may also be used to inform drivers of the current autonomy level of the vehicle.

We have analysed data about driver Situation Awareness (SA) detection systems, classified equipment and approaches to measure SA, and reviewed all accidents of highly-automated vehicles in California, USA. The outcomes will define scenarios that will need to be designed to validate new technologies for highly-automated vehicles. Also, it will guide research and development in other sectors such as manufacturing industry where SA is critical.
Sectors Digital/Communication/Information Technologies (including Software),Education,Healthcare,Transport

 
Description Contribution to the UK Connected and Automated Mobility Roadmap to 2030
Geographic Reach National 
Policy Influence Type Participation in a advisory committee
URL https://zenzic.io/roadmap/
 
Description Expert for new funding programme for the Federal Ministry of Education and Research (Germany)
Geographic Reach National 
Policy Influence Type Participation in a advisory committee
Impact n/a
 
Description Capital Award in Support of Early Career Researchers at UCL
Amount £425,000 (GBP)
Funding ID EP/S01800X/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 09/2018 
End 03/2020
 
Description DfT Transport-Technology Research Innovation (T-TRIG) Grant - GAIN
Amount £2,995,859 (GBP)
Organisation Department of Transport 
Sector Public
Country United Kingdom
Start 01/2020 
End 06/2020
 
Description EIT Urban Mobility - AI-TraWell
Amount € 401,463 (EUR)
Funding ID 128547 
Organisation European Institute of Innovation and Technology (EIT) 
Sector Public
Country Hungary
Start 01/2020 
End 12/2020
 
Description Performing Place - Trellis programme - EPSRC Impact Acceleration Account - The Dancing Brain
Amount £8,000 (GBP)
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 01/2020 
End 05/2020
 
Description Two Departmental PhD funding
Amount £140,000 (GBP)
Organisation University College London 
Sector Academic/University
Country United Kingdom
Start 03/2020 
 
Description UK-RAS Strategic Task Group for Soft Robotics
Amount £38,500 (GBP)
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Department UK-RAS Network
Sector Public
Country United Kingdom
Start 02/2020 
End 02/2020
 
Title An analytical methodologyto identify EEG signatures associated with SA on various regions of the brain 
Description An important aspect in safety-critical domains is Situational Awareness (SA) where operators consolidate data into an understanding of the situation that needs to be updated dynamically as the situation changes over time. Among existing measures of SA, only physiological measures can assess the cognitive processes associated with SA in real-time. Some studies showed promise in detecting cognitive states associated with SA in complex tasks using brain signals (e.g. electroencephalogram/EEG). We developed an analytical methodology to identify EEG signatures associated with SA on various regions of the brain. 
Type Of Material Physiological assessment or outcome measure 
Year Produced 2021 
Provided To Others? Yes  
Impact How to identify operator's situational awareness is one of the most important problems in ergonomics study. Although there are several methods (SAGAT, 3D-SART, SPAM, operation performance, et al.) which can be used for measuring operator's situational awareness, the physiological index which can clearly reflect operator's situational awareness is hard to find, especially for the level 2 and level 3 of situational awareness. We investigated the correlation between situational awareness and EEG signals. 
URL https://www.sciencedirect.com/science/article/pii/S0925231220319263
 
Title Hybrid fluidic actuation system for a foam-based soft actuator 
Description Actuation means for soft robotic structures are manifold: despite actuation mechanisms such as tendon-driven manipulators or shape memory alloys, the majority of soft robotic actuators are fluidically actuated - either purely by positive or negative air pressure or by hydraulic actuation only. The novel idea of employing hybrid fluidic - hydraulic and pneumatic - actuation for soft robotic systems is proposed in this project. The concept and design of the hybrid actuation system as well as the fabrication of the soft actuator are: Polyvinyl Alcohol (PVA) foam is embedded inside a casted, reinforced silicone chamber. A hydraulic and pneumatic robotic syringe pump are connected to the base and top of the soft actuator. We found that a higher percentage of hydraulics resulted in a higher output force. Hydraulic actuation further is able to change displacements at a higher rate compared to pneumatic actuation. Changing between Hydraulic:Pneumatic (HP) ratios shows how stiffness properties of a soft actuator can be varied. 
Type Of Material Technology assay or reagent 
Year Produced 2020 
Provided To Others? Yes  
Impact Contribution to fundamental science in soft robotic actuation presented at IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 
URL https://www.youtube.com/watch?v=Ipa3_-FVvHw
 
Title 32-channel dry-EEG dataset from participants completing the Situational Awareness test in the Psychology Experiment Building Language 
Description A new data set from 32 participants completing the Situational Awareness test in the Psychology Experiment Building Language (PEBL) is collected using a 32-channel dry-EEG headset. 
Type Of Material Database/Collection of data 
Year Produced 2021 
Provided To Others? Yes  
Impact Multiple collaboration opportunities with Industry and Academia 
URL https://rdr.ucl.ac.uk/
 
Description In-kind contribution of a car 
Organisation Honda Of The UK Manufacturing Ltd
Country United Kingdom 
Sector Private 
PI Contribution ideas/knowledge
Collaborator Contribution An in-kind contribution of a Honda HRV to UCL
Impact - Building a stationary Driver-in-the-Loop car simulator (Sigma series) at UCL in collaboration with Ansible Motion Ltd (Collaboration between Mechanical Eng. and Transport Eng.) - Successful funding application: Securing another research grant from DfT Search Transport-Technology Research Innovation Grants ((Collaboration between Mechanical Eng., Transport Eng. and Human factors psychology, Computer Science)
Start Year 2019
 
Description Knowledge exchange with Ansible Motion Ltd 
Organisation Ansible Motion
Country United Kingdom 
Sector Private 
PI Contribution Ideas and knowledge exchange especially on: - Building a simulation environment using Unity - A robotic seat to provide haptic feedback to drivers
Collaborator Contribution Ideas and knowledge on the current practice and future avenues of Driver-in-the Loop simulators around the world
Impact Collaboration Successful funding application Submission of a new proposal to UKRI Trustworthy Autonomous Systems Hub for collaboration on a specific topic
Start Year 2018
 
Description Knowledge exchange with McLaren Applied 
Organisation McLaren Applied Technologies
Country United Kingdom 
Sector Private 
PI Contribution We provided our knowledge domain expertise and knowledge.
Collaborator Contribution They have expertise in data analytics to provide decision insight in complex scenarios, improve decision making. They provided their knowledge on data management tools.
Impact Submission of a proposal to UKRI Trustworthy Autonomous Systems Hub for collaboration on a specific topic.
Start Year 2021
 
Description Public engagement in collaboration with Akamedi 
Organisation Akademi
Country United Kingdom 
Sector Charity/Non Profit 
PI Contribution Knowledge, ideas and equipment
Collaborator Contribution Contacts, knowledge and ideas
Impact Public engagement and outreach Successful funding application: The Dancing Brain funded by EPSRC Impact Acceleration Account
Start Year 2019
 
Description Research collaboration with MIRA Ltd 
Organisation MIRA Ltd
Country United Kingdom 
Sector Private 
PI Contribution We provided use of equipment and models.
Collaborator Contribution They provided knowledge and collaborative research ideas. They will us with the experimental design, interpretation of results, scale-up using their CAV testing equipment and facilities such as their Network Guided Vehicle/City Circuit.
Impact Potential collaboration: - Identified potential joint PhD studies - Joint proposal writing - Submission of a new proposal to UKRI Trustworthy Autonomous Systems Hub for collaboration on a specific topic
Start Year 2019
 
Description 11 Invited Talks (7) and Panel Discussions (4) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Dissemination, training, knowledge exchange, and contribution to the UK Connected and Automated Mobility Roadmap to 2030 :

1. Invited as expert on intelligent mobility for panel discussion with Amey, TfL and Connected Places Catapult (70 innovation leaders) / Title: Intelligent Mobility - What needs to happen for sustainable transport / Location: Digital Cities Networking Event - The Drive to Sustainability, EIT Digital, London, UK
2. Invited as expert for panel discussion with FiveAI, Ford Mobility and Innovate UK (50 CAV experts) / Title: Society and people / Location: Workshop on the UK Connected and Automated Roadmap to 2030, Zenzic (formerly known as Meridian), London, UK
3. Invited as expert for panel discussion with Nissan, Waze and RAC (40 CAV experts) / Title: Connectivity: How can the connected car contribute to a better London? / Location: The London Automotive Forum, Knowledge Transfer Network (KTN) & Transport for London (TfL), London, UK
4. Invited as expert for panel discussion with DfT and TfL (100 international students) / Title: Integration of Connected and Autonomous Vehicles within Transport Systems / Location: Global Student Engineering Conference 2017 - Sustainable Cities, UCL Mechanical Engineering, London, UK
5. Invited online talk (200 experts from academia, industries and cities) / Title: Morphing between Levels of Situational Awareness in Highly-automated Vehicles / Smart Cities, IoT and AI, AI Tech North 2020, UK
6. Invited talk (80 international students and academics) / Title: Overview of the Development of Technology of CAV and Drones / Location: Annual Leadership Training Programme, University of Oxford, Oxford, UK
7. Invited talk (80 international students and academics) / Title: Implementation and Impact of CAV Drones for Cities / Location: Annual Leadership Training Programme, University of Oxford, Oxford, UK
8. Invited talk along with Stanford, MIT and University of Cambridge (85 academics) / Title: Morphing between Levels of Situational Awareness in Highly-automated Vehicles / Location: International Conference on Robotics and Automation (ICRA) Workshop on Soft Haptic Interaction, Montreal, Canada
9. Invited seminar (15 academics) / Title: Intuitive Haptic Feedback for Take Over Tasks in AVs / Location: UCL Energy Institute, London, UK
10. Invited talk along with Stanford, MIT, and EPFL and Technical University of Munich / Title: Human-machine Interaction for autonomous vehicles / Location: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) Workshop on Shape Changing Robotic Structures and Interfaces, Madrid, Spain
11. Invited talk (70 academics) / Title: Future of Mobility: Modelling and Optimisation / Location: The University of Sydney, Sydney, Australia
Year(s) Of Engagement Activity 2018,2019,2020
 
Description Four Invited Panels 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact 1. Accelerating CAV deployment - Test-beds, Pilots and Demonstrators (IDEALondon/Capital Enterprise)
2. Digital Cities Networking Event - The Drive to Sustainability (EIT Digital)
3. Workshop on the UK Connected and Automated Roadmap to 2030 (Zenzic (formerly known as Meridian))
4. The London Automotive Forum (KTN & TfL)
Year(s) Of Engagement Activity 2019,2020
 
Description Invited talk - Inherently Soft - Inherently Safe: Robotics Research at UCL, University of Toronto Robotics Institute, Canada 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact n/a
Year(s) Of Engagement Activity 2019
 
Description Invited talk - Morphing between Levels of Situational Awareness in Highly-automated Vehicles, ICRA 2019 Workshop on Soft Haptic Interaction: Modelling, Design and Application, Montreal, Canada 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact n/a
Year(s) Of Engagement Activity 2019
 
Description Invited talk - Soft Material Robotics: Inherently Soft - Inherently Safe, Brunel Robotics Engineering Society, Brunel University, London, UK 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Undergraduate students
Results and Impact n/a
Year(s) Of Engagement Activity 2019
 
Description Keynote Talk 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Industry/Business
Results and Impact Trends in Urbanization, Smart Cities and Transport (BMW Summer School)
Year(s) Of Engagement Activity 2019