Autonomous vehicles for wheelchair use

Lead Research Organisation: Imperial College London
Department Name: Dept of Computing

Abstract

To integrate current methods for autonomous vehicles into an indoor-use wheelchair for paraplegic users. This base system is going to be controlled by a gaze-based interface that decodes high-level action intentions (where a user would like to go) to take him there.

Publications

10 25 50

Studentship Projects

Project Reference Relationship Related To Start End Student Name
EP/N509486/1 01/10/2016 31/03/2022
1979819 Studentship EP/N509486/1 01/10/2017 31/07/2021 Mahendran Subramanian
 
Description People with severe lower-limb movement impairments rely heavily on devices such as wheelchairs for their daily activities. Electric wheelchair controllers such as joysticks, sip and puff, or head-mount gyroscope-based devices are effective when giving full control of the wheelchair to the user, but require continuous attention, and engage them on a low-level of control. We propose the use of natural gaze patterns as an implicit interface for high-level wheelchair command, while low- level control, including navigation and obstacle avoidance is done autonomously. In this manner, the system allows subjects to navigate their wheelchairs in dynamic environments, without the need to interact with a controller, engaging them instead on a higher cognitive level. Wearable eye-trackers are used to record eye-movements, with online object detection and labelling, to identify the object of interest, and regions of gaze interest within the object boundaries. Resulting data from contextual gaze fixation duration and gaze length are used to build an intention decoding classifier. This, combined with autonomous navigation algorithms on the wheelchair platform, allow users to accomplish their navigation requirements, as indicated naturally by their gaze, in an autonomous manner.
Exploitation Route Cognitive technologies towards smart vehicles are evolving, but specific to wheelchairs are often overlooked. High- level gaze informatics can be complemented with context-aware algorithms for natural interaction with the environment through robotic devices or autonomous vehicles. Herein, we harvest the eye movements to enhance the cognitive abilities of an autonomous wheelchair platform as eye movements are correlated with motor intention and act as a precursor to movement. First, we developed tools to estimate 3D gaze point and to recognise the object at the wheelchair user's 'Area Of Interest' for high-level intention decoding. This 3D eye-tracking tool with practical accuracy levels to analyse natural human behaviour during physical interaction with the environment was used to obtain human intention in autonomous systems for natural, user-centric interaction with the environment. With high-level intention decoding capability, such cognitive wheelchair systems can not only perform autonomous mobility tasks but also function in a contextual, semantic and continuous spectrum of tasks.
Sectors Aerospace, Defence and Marine,Digital/Communication/Information Technologies (including Software),Education,Electronics,Healthcare,Leisure Activities, including Sports, Recreation and Tourism,Manufacturing, including Industrial Biotechology,Transport

 
Description Nokia corporation is interested in commercializing our following inventions. They have already paid for the patent applications, and we have signed agreements with them to take these technologies forward 1. Mahendran Subramanian, Aldo Faisal (2017). Breath detection extraction from motion data. (Nokia Corporation - Imperial Business School - School of Bioengineering Project) EP18213990.7 2. Mahendran Subramanian, Aldo Faisal (2017). Movement indication for a scene based on video data. (Nokia Corporation - Imperial Business School - School of Bioengineering Project) EP18213981.6 Numerous investors have also shown interest in investing and commercializing our 3D gaze-based wheelchair technology - Apart from EPSRC my PhD topic also received some external funding over the course based on the outcome from commercial companies like Toyota, BMW, Nokia, which shows the real-world impact of the topic. We have already started the process of developing our technology into a commercial project.
First Year Of Impact 2017
Sector Digital/Communication/Information Technologies (including Software),Healthcare
Impact Types Societal,Economic,Policy & public services

 
Description Demo of my AI wheelchair at the Imperial Global Science Policy Forum - AI Showcase
Geographic Reach Multiple continents/international 
Policy Influence Type Participation in a advisory committee
Impact I performed a Demo on Autonomous (AI) Wheelchair (resulting from my PhD- 3d gaze based semi-autonoous wheelchair) Autonomous Urban Mobility for the elderly and the severly disabled. The demo was perfromed to politicians/ policymakers from throughout the globe including UK politicians/ policymakers.
 
Description Bay France - BMW - TUM grant to attend Summer School on Intelligent Vehicles
Amount £600 (GBP)
Organisation Bayerische Motoren Werke (BMW) 
Sector Academic/University
Country Germany
Start 06/2018 
End 08/2018
 
Description Faculty of Engineering - Travel grant
Amount £1,000 (GBP)
Organisation Imperial College London 
Sector Academic/University
Country United Kingdom
Start 06/2018 
End 08/2018
 
Description Nanotherics Research and Development Fund
Amount £150,000 (GBP)
Organisation nanoTherics Ltd 
Sector Private
Country United Kingdom
Start 11/2010 
End 09/2019
 
Description Nokia Corporation - Imperial Business School initiative
Amount £40,000 (GBP)
Organisation Nokia 
Sector Private
Country Global
Start 07/2017 
End 05/2021
 
Description Tech 4 Good - Inclusive Design Award (Finalist)
Amount £710 (GBP)
Organisation BT Group 
Sector Private
Country United Kingdom
Start 07/2019 
 
Description Toyota - Mobility Unlimited Challenge Discovery Award
Amount $50,000 (USD)
Organisation Toyota Motor Corporation 
Sector Private
Country Japan
Start 04/2018 
End 09/2018
 
Title Alternating magnetic field mediated drug release testing setup 
Description Alternating magnetic field mediated drug release testing setup (finished, published in Applied Physics A and undergone external evaluation in IIT, Italy): Novelty - Controlled drug delivery for magnetic fluid cancer hyperthermia experiments. Heat dissipation during magnetization reversal processes in magnetic nanoparticles (MNP), upon exposure to alternating magnetic fields (AMF), has been extensively studied in relation to applications in magnetic fluid hyperthermia (MFH). The paper demonstrates the design, fabrication, and evaluation of a drug release analysis system (DRAS); and (iii) a high flux density module for AMF-mediated MNP studies. The proposed DRAS is demonstrated by an AMF-mediated drug release proof-of-principle experiment. Numerical calculations estimating the absolute and root mean squared flux densities, and other MNP - AMF studies suggest that the proposed stacked planar coil module could be employed for calorimetry. 
IP Reference  
Protection Protection not required
Year Protection Granted
Licensed Commercial In Confidence
Impact Device used for in vivo testing of controlled drug delivery for magnetic fluid cancer hyperthermia experiments. Currently sold by nanotherics Ltd (UK based company that funded this research) and bought globally (£ 3500-4000 GBP).
 
Title Breath detection extraction from motion data. 
Description Mahendran Subramanian, Aldo Faisal (2017). Breath detection extraction from motion data. (Nokia Corporation - Imperial Business School - School of Bioengineering Project) EP18213990.7 
IP Reference EP18213990.7 
Protection Patent application published
Year Protection Granted 2017
Licensed Commercial In Confidence
Impact The research was funded by Nokia Corporation
 
Title Method and apparatus to enable real time non-contact in vitro temperature measurement for nanoparticle calorimetry in time varying magnetic field 
Description An apparatus for maintaining organic cells in vitro whilst simultaneously applying a time-varying radio frequency magnetic field and taking non-contact, real-time temperature or calorimetry measurements. The organic cells are cultured with magnetically susceptible particles or nanoparticles so that hyperthermia takes place within the magnetic field. The apparatus comprises: an incubation chamber; a magnetic field source, e.g. a coil or gapped core, connected to an alternating magnetic field generator; a cell culture vessel preferably with a biocompatible base of thickness 50µm or less; temperature sensors, preferably either a fibre optic or an infrared sensor, whose tips are attached/focussed to the underside of the cell culture vessel respectively. The magnetic field source may be cooled e.g. using liquid nitrogen or water. The magnetically susceptible particles may comprise gold. The temperature sensor may be of single channel or multichannel, may contain a computer interface and software to measure and record the real-time temperature change. 
IP Reference GB2551743 
Protection Patent application published
Year Protection Granted 2018
Licensed Commercial In Confidence
Impact Nanomagnet actuating air core inductor with time lapse imaging setup (finished, published in International Journal of Hyperthermia and currently patent pending): Novelty - Controlled drug delivery, magnetic fluid cancer hyperthermia experiments can be monitored real time, when the cells/ tissue samples are exposed to alternating magnetic field. Currently sold by nanotherics Ltd (UK based company that funded this research) and bought globally (£ 3500-4000 GBP), including the top 20 (world rank) Universities.
 
Title Movement indication for a scene based on video data 
Description Mahendran Subramanian, Aldo Faisal (2017). Movement indication for a scene based on video data. (Nokia Corporation - Imperial Business School - School of Bioengineering Project) EP18213981.6 
IP Reference EP18213981.6 
Protection Patent application published
Year Protection Granted 2017
Licensed Commercial In Confidence
Impact The research was funded by the Nokia Corporation
 
Title Self-driving AI wheelchair system with gaze based intention decoding capability 
Description Self-driving AI wheelchair system (PhD project): Novelty - Eye gaze-controlled, artificially intelligent semi-autonomous wheelchair capable of decoding human intentions from their eye movements. Won Discovery award - $50,000 USD. The prize was announced as part of Mobility Unlimited Challenge Discovery Award, which is funded by Toyota Mobility Foundation and run by Nesta's Challenge Prize Centre. Featured on the Imperial College London homepage. Have built an Autonomous Wheelchair with a Novel eye gaze interface and the submission came on the Top 10 for Top 5 - $500,000 USD prize - Toyota Mobility Final Challenge: Finalist Tech4Good award (Inclusive design award) - BT, UK. 
Type Therapeutic Intervention - Medical Devices
Current Stage Of Development Initial development
Year Development Stage Completed 2019
Development Status Under active development/distribution
Impact Design and development were published in IEEE EMBC 2019 and IEEE IROS 2019 - Towards Cognitive Vehicles Workshop This technology will help improve independent mobility and quality of life for the severely disabled. 
 
Company Name FARADAY-FLEMING LABORATORY LTD 
Description We have set up this company in order to commercialize our Biomedical Engineering products. Currently, we have attracted £710, i.e. the prize money we got for being a Tech4good UK - Inclusive design award finalist for my (Mahendran Subramanian) PhD topic - 3D gaze-based semi-autonomous wheelchair. We are currently working towards attracting more money from external funding sources, including venture capital investments. 
Year Established 2017 
Impact We intend to commercialize the final product that I have developed based on my PhD research: 3D gaze-based semi-autonomous wheelchair through this company. The technology is also capable of gaze-based intention decoding for goal definition.
 
Description BMW Summer School - Emotional Aware Assistance for Intelligent Vehicles. Technical University Munich, Germany 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Industry/Business
Results and Impact Mahendran Subramanian, Noyan Songur, Darrell Adjei, Aldo Faisal (2018). A.Eye-Drive: Autonomous
navigation for the ageing and physically impaired. BMW Summer School - Emotional Aware Assistance
for Intelligent Vehicles. Technical University Munich, Germany.

Noyan Songur+, Mahendran Subramanian+, Aldo Faisal (2018). A.Eye-Drive: Gaze-based intention
decoding for human-in-the-loop autonomy. BMW Summer School - Emotional Aware Assistance for
Intelligent Vehicles. Technical University Munich, Germany.

[Outcome: Best Poster Award ]
Year(s) Of Engagement Activity 2018
 
Description Conference - Understanding images in biological and computer vision. Royal Society of London, UK. 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact Mahendran Subramanian, Lou-Ann Raymond, Margherita Piccini, Pavel Orlov, Aldo Faisal (2018) Decoding
natural eye movements during driving to steer a wheelchair. Conference - Understanding images in
biological and computer vision. Royal Society of London, UK.
Year(s) Of Engagement Activity 2017
 
Description Demonstration of our 3D gaze based AI wheelchair at The AI Summit, TechXLR8 2019, London, UK 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Policymakers/politicians
Results and Impact TechXLR8 encompasses 7 of the biggest technology conferences under one roof including The AI Summit, Blockchain for Business, IoT World Europe, Cloud & DevOps World, Startup Elevate, and AR & VR World.
TechXLR8 is London Tech Week's flagship event, Europe's leading tech festival that welcomes 55,000 attendees across a week filled with creativity, talent and innovation. This 2- day tech festival ignites transformation through connecting 15,000 attendees with industry leaders, iconic brands and innovative products.

I demonstrated my invention, i.e. 3D gaze-based semi-autonomous wheelchair to more than 100 event participants who were interested in our technology. We also did a lot of Networking and Interacting directly to the whole community. I got the opportunity to be amongst the latest cutting-edge technology, hear from 600+ tech guru speakers and experience impressive demo zones which will help elevate my AI knowledge first hand. I was also interviewed by TV channels at this event.
Year(s) Of Engagement Activity 2019
 
Description Done numerous demonstrations of our 3D gaze based AI wheelchair 
Form Of Engagement Activity Participation in an open day or visit at my research institution
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact Done numerous demonstrations of our 3D gaze based AI wheelchair to visitors at our research facility, TV channels incl. BBC, ITV, Channel 4, and on conferences held within Imperial College London on robotics and AI
Year(s) Of Engagement Activity 2017,2018,2019,2020
 
Description Google Poster Competition 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Postgraduate students
Results and Impact Mahendran Subramanian, Darrell Adjei, Aldo Faisal (2018). 3D Gaze based Semi-Autonomous Wheelchair. Google Poster Competition. Department of Computing, Imperial College, UK.

[Outcome - Best Poster Award].
Year(s) Of Engagement Activity 2018
 
Description Patient house visit for Demonstration and evaluation of my invention 3D gaze based semi-autonomous wheelchair - Co creation 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Patients, carers and/or patient groups
Results and Impact Drove my invention (3D gaze-based semi-autonomous wheelchair) to severely disabled patient homes (Peterborough and Oxfordshire) to perform demonstration and evaluation of our technology along with their participation, i.e. Co-creation. This helped me to understand more of their needs and how our technology can be modified or improved to well suit its purpose. Their enthusiasm and results ensured that the technology is capable of improving patient quality of life as well as provide independency to the severely disabled and the elderly.
Year(s) Of Engagement Activity 2018
 
Description Poster presentation IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2019); Workshop - Towards Cognitive Vehicles Workshop. 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Mahendran Subramanian, Suhyung Park, Pavel Orlov, Aldo Faisal (2019). Cognitive Wheelchair: A Personal Mobility Platform. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS
2019); Workshop - Towards Cognitive Vehicles Workshop.
Year(s) Of Engagement Activity 2019
 
Description Poster presentation on Conference - Interactive eye gaze 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact Mahendran Subramanian, Lou-Ann Raymond, Margherita Piccini, Pavel Orlov, Aldo Faisal (2018) Natural
Gaze data driven wheelchair. Conference - Interactive eye gaze. University College London, UK.
Year(s) Of Engagement Activity 2017