Sensorimotor Learning for Control of Prosthetic Limbs

Lead Research Organisation: Newcastle University
Department Name: Sch of Engineering

Abstract

Worldwide, there are over three million people living with upper-limb loss. Recent wars, industrialisation in developing countries and vascular disease, e.g. diabetes, have caused the number of amputations to soar. Adding to this population each year, one in every 2,500 people are born with upper-limb reduction. Advanced prostheses can play a major role in enhancing the quality of life for people with upper-limb loss, however, they are not available under the NHS. Notably, many people with traumatic limb loss are otherwise physically fit. If they are equipped with advanced prostheses and treated to recover psychologically, they can live independently, with minimal need for social support, return to work and contribute to the economy.

There are a plethora of underlying reasons that limit wide clinical adoption of advanced prosthetic hands. For instance, surveys on their use reveal that 20% of upper-limb amputees abandon their prosthesis, with the primary reason being that the control of these systems is still limited to one or two movements. In addition, the process of switching a prosthetic hand into an appropriate grip mode, e.g. to use scissors, is cumbersome or requires an ad-hoc solution, such as using a smart phone application. Other reasons include: users finding their prosthesis uncomfortable or unsuitable for their needs. As such, everyday tasks, such as tying shoe-laces, are currently very challenging for prosthetic hand users. These functional shortcomings, coupled with high costs and lack of concrete evidence for added benefit, have emerged as substantial barriers limiting clinical adoption of advanced prosthetic hands.

The long-term aim of this cross-disciplinary programme is to develop, and move towards making available, the next generation of prosthetic hands that can improve the users' quality of life. Our underlying scientific novelty is in utilising users' capability of learning to operate a prosthesis. For instance, we examine the extent to which the activity of muscles can deviate from natural patterns employed in controlling movement of the biological arm and hand and whether prosthesis users can learn to synthesise these functional maps between muscles and prosthetic digits. Basing this approach upon our pilot data, we hypothesise that practice and availability of sensory feedback can accelerate this learning experience. To address this fundamental question, we will employ in-vivo experiments, exploratory studies involving able-bodied volunteers and pre-clinical work with people with limb loss. The insight gained from these studies will inform the design of novel algorithms to enable seamless control of prosthetic hands. Finally, the programme will culminate with a unifying theory for learning to control prosthetic hands that will be tested in an NHS-approved, pre-clinical trial.

Maturing this approach into a clinically-viable solution needs a dedicated team of engineers and scientists as well as a consortium of users, NHS-based clinicians and healthcare and high-tech industries. With the flexibility that a Healthcare Technologies Challenge Award affords me, I will be able to nurture and grow sustainably my multi-disciplinary team. In addition, this flexible funding will enable to focus on a converging research programme with the ultimate aim of providing prosthetic solutions that enhance NHS-approved clinical patient outcome measures significantly.

Within this programme, I will identify and bring together the engineering, scientific, clinical, ethical and regulatory elements necessary to form a recognised national hub for the development of next-generation prosthetics. This work will provide the foundations for my 15-year plan to establish the Centre for Bionic Limbs. The origin of this Centre will be to act as a mechanism to safeguard engineering and scientific innovations, increase value, and accelerate transfer into commercial and clinical fields.

Planned Impact

This programme aims at development of a new biologically-informed approach, termed Pattern Learning, for the control of upper-limb hand prostheses. The impact of this programme is as the following:

Society: Loss of the hand, and the complexities it entails, is one of the most feared conditions having a large impact upon individuals, their families and the society. In addition to the loss of function that losing a limb causes, phantom pain and psychological distress can be severe. Life-time care requires taking drugs that have side effects. Use of advanced prostheses can improve the quality of life of users dramatically and contribute to their personal dignity, independence and more effective inclusion in the society. As such, prostheses that offer seamless control of multi-joint movements are essential to minimise the effect of limb loss.

The challenge to provide effective prostheses is a recurrent feature of comments from users in Patient and Public Involvement meetings. In addition, It has been a key theme in recent national reports on prosthetic services in the UK (e.g. Chavasse report, 2014) particularly in relation to injured service personnel and survivors of major trauma. These issues are discussed regularly during meetings of the All-Party Parliamentary Limb Loss Group. The proposed programme has the potential to make a step change in the quality of control of prosthetic hands.

In addition to prosthetics, the knowledge and technologies, e.g. stimulation of the peripheral nerves, that we will develop in this programme can be translated to other movement disorders such as foot drop, a muscular weakness or paralysis that makes walking difficult because the patient cannot lift the front part of the foot and toes.

Economy: The results of this project will cement the UK's reputation as the leading country for prosthetic design and manufacturing. Three of the four major manufacturers of upper- and lower-limb prosthetic limbs are based in the UK. The market for advanced prostheses and bionics is growing rapidly. This project will provide significant know-how and novel technologies to help the UK's prosthetics industry remain competitive in a global market. Financial security of the British upper-limb prosthetics industry will help to reduce the production and R&D cost allowing the NHS to cover the cost for a larger number of patients. Moreover, able-bodied individuals who want advanced interfaces to better operate or interact with robots or games can also benefit from the results of this work. In addition, the implantable electronic technologies for neural stimulation that we develop can be taken up by the Electronic Medicine (electroceuticals) industry and used for treatment of a range of health conditions such as diabetes and depression.

Knowledge: Understanding the flexibility and limits of the human motor systems in learning new skills and their application in prosthesis control are very exciting and timely, as evident by the number recent publications in Nature on these topics. Our cross-species approach will help development of a unifying theory for Pattern Learning which is very exciting scientifically because with Pattern Learning we will be able to develop a platform for truly plug and play prosthetic systems.

People: This programme will provide me with a platform upon which I can work towards becoming a research leader in the field of prosthetics. In addition, it will provide high-quality training in the general areas of signal analysis and sensorimotor control for three research associates (RA1-3) and several PhD students. All team members will gain knowledge and expertise through the process of solving challenging problems. They will spend time at both Newcastle University's School of Electrical and Electronic Engineering and Institute of Neuroscience as well as at collaborating hospitals (Freeman and Salford Royal). They will benefit from a truly multi-disciplinary research environment.

Publications

10 25 50

publication icon
Alameer A (2020) Objects and scenes classification with selective use of central and peripheral image content in Journal of Visual Communication and Image Representation

publication icon
Blana D (2020) Model-Based Control of Individual Finger Movements for Prosthetic Hand Function. in IEEE transactions on neural systems and rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society

publication icon
Brunton E (2018) Recovery of the Response of Sensory Fibers to the Second of a Pair of Peripheral Nerve Stimuli. in Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual International Conference

publication icon
Brunton EK (2019) Temporal Modulation of the Response of Sensory Fibers to Paired-Pulse Stimulation. in IEEE transactions on neural systems and rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society

publication icon
Dyson M (2018) Data Driven Spatial Filtering Can Enhance Abstract Myoelectric Control in Amputees. in Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual International Conference

publication icon
Dyson M (2018) Abstract Decoding using Bayesian Muscle Activation Estimators. in Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual International Conference

publication icon
Dyson M (2020) Learning, Generalization, and Scalability of Abstract Myoelectric Control. in IEEE transactions on neural systems and rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society

 
Title Supplementary video from IoT for beyond-the-laboratory prosthetics research 
Description Video clips for online data collection, device monitoring and user feedback submission during the pick and place experiment 
Type Of Art Film/Video/Animation 
Year Produced 2022 
URL https://rs.figshare.com/articles/media/Supplementary_video_from_IoT_for_beyond-the-laboratory_prosth...
 
Title Supplementary video from IoT for beyond-the-laboratory prosthetics research 
Description Video clips for online data collection, device monitoring and user feedback submission during the pick and place experiment 
Type Of Art Film/Video/Animation 
Year Produced 2022 
URL https://rs.figshare.com/articles/media/Supplementary_video_from_IoT_for_beyond-the-laboratory_prosth...
 
Description When controlling a prosthesis, the patterns of muscular activity can differ from those used to control the biological limbs.
Exploitation Route 2019 The above finding can radically change the way we look at controlling prosthetic limbs. We are at an early stage of collaborating with a company to embed this approach in their controller.
Sectors Electronics

Healthcare

 
Description 2019 This is a 5-year programme. The main impact in Year 1 has been on the engagement of the public with STEM and prosthetics in particular
First Year Of Impact 2018
Sector Electronics,Healthcare
Impact Types Societal

 
Description Animal Free UK
Amount £2,000 (GBP)
Organisation Animal Free Research UK 
Sector Charity/Non Profit
Country United Kingdom
Start 05/2019 
End 08/2019
 
Description Creation of a User-Centred Vision for Prosthetic Limbs and Clinical Care
Amount £250 (GBP)
Organisation Port-ER 
Sector Charity/Non Profit
Country United Kingdom
Start 03/2018 
End 03/2019
 
Description RISE-WELL Critical solutions for elderly well-being
Amount € 2,691,279 (EUR)
Funding ID DLV-860173 
Organisation European Commission H2020 
Sector Public
Country Belgium
Start 09/2019 
End 09/2023
 
Description Starworks Child Prosthetics
Amount £47,790 (GBP)
Funding ID STWK-006 
Organisation National Institute for Health Research 
Department NIHR Devices for Dignity Healthcare Technology Co-Operative
Sector Public
Country United Kingdom
Start 03/2018 
End 12/2018
 
Description UKIERI
Amount £12,000 (GBP)
Organisation UK-India Education and Research Initiative (UKIERI) 
Sector Academic/University
Country United Kingdom
Start 08/2019 
End 08/2021
 
Title pyEMG 
Description Python package for offline and real-time myoelectric control by using sEMG and IMU signals 
Type Of Material Physiological assessment or outcome measure 
Year Produced 2018 
Provided To Others? Yes  
Impact The impact is growing, as evidenced by new downloads 
URL https://github.com/agamemnonc/pyEMG
 
Title Data for 'IoT for beyond-the-laboratory prosthetics research' 
Description The original data for the pick and place experiment and the MATLAB script to plot the data 
Type Of Material Database/Collection of data 
Year Produced 2022 
Provided To Others? Yes  
URL https://rs.figshare.com/articles/dataset/Data_for_IoT_for_beyond-the-laboratory_prosthetics_research...
 
Title Data for 'IoT for beyond-the-laboratory prosthetics research' 
Description The original data for the pick and place experiment and the MATLAB script to plot the data 
Type Of Material Database/Collection of data 
Year Produced 2022 
Provided To Others? Yes  
URL https://rs.figshare.com/articles/dataset/Data_for_IoT_for_beyond-the-laboratory_prosthetics_research...
 
Title ENG recordings in response to mechanical stimulation 
Description Electroneurogram signals recorded from 16-channel cuff electrodes placed on the sciatic nerve of Sprague Dawley rats. The recordings were made in response to three types of mechanical stimulation, namely, proprioception, touch and nociception. The data files include synchronisation comments and signals that indicate when a stimulus was applied and what the stimulus was. Proprioceptive responses were recorded in response to a servo motor moving the foot to one of 6 different angles. Touch responses were recorded in response to heel of the foot being touched with one of two Von Frey fibres, corresponding to a force of either 100 or 300 grams. Finally, Nociception responses were recorded in response to the foot being pinched using a pair of forceps either on the heel, or the outer toe. 
Type Of Material Database/Collection of data 
Year Produced 2017 
Provided To Others? Yes  
Impact Experiments with animals create large amount valuable data on which various algorithms can be applied. This is a first database of its kind which made available online. We hope it will help with the reduction of number of animals used in research. 
URL https://rdm.ncl.ac.uk/landing/pages/10.17634/141353-2
 
Title Online Codebase 
Description Intelligent Sensing Github 
Type Of Material Computer model/algorithm 
Year Produced 2019 
Provided To Others? Yes  
Impact several downloads by others in the field 
URL https://github.com/intellsensing
 
Company Name Neuranics 
Description Neuranics develops magnetic neurotechnology intended to integrate with wearables and implantable devices. 
Year Established 2021 
Impact Early stage but so far the company created 11 jobs
Website https://neuranics.com/
 
Description A patient public involvement event 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Other audiences
Results and Impact In December 2018, the Intelligent Sensing laboratory at Newcastle University hosted a one-day workshop to capture challenges and opportunities to improve the lives of people living with limb loss. This was a collaborative project that aimed to create a user-centred vision for prosthetic devices and clinical care. In order to achieve this aim, the project team required interaction with a variety of different stakeholder. Therefore the workshop was developed to benefit academics, NHS clinicians, industry specialists¬, charity executives and prosthesis users; all of which were in attendance at the workshop.
Year(s) Of Engagement Activity 2018
 
Description A talk presented at Sheffiled University 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact The CATCH Network learned about machine learning in the control of prosthetic hands and Patient and Public Involvement. We were delighted to be joined by not one, not two, but three exciting speakers at the June 2016 CATCH Networking Lunch.
Year(s) Of Engagement Activity 2017
URL http://www.catch.org.uk/news-articles/6th-networking-lunch/
 
Description Future prosthetic: towards the bionic human 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact Kia Nazarpour gave an interview to The Engineer Magazine about the science-fiction vision of robotic prosthetic limbs that can be controlled by the brain and provide sensory feedback, which in his opinion is coming closer to reality.
Year(s) Of Engagement Activity 2018
URL https://www.theengineer.co.uk/future-prosthetic/
 
Description Interview with Dazed 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact Interview about "How AI could increase art world accessibility for disabled artists"
Year(s) Of Engagement Activity 2018
URL https://www.dazeddigital.com/art-photography/article/41334/1/how-ai-could-increase-art-world-accessi...
 
Description Interview with Verdict 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact This was an interview about how AI is exploding into healthcare
Year(s) Of Engagement Activity 2018
URL https://www.verdict.co.uk/ai-healthcare-growth-apps/
 
Description Royal Society Exhibition Blog 
Form Of Engagement Activity Engagement focused website, blog or social media channel
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact We have been selected by the Royal Society to exhibit our work at their Summer Science FEstival, attracting around 14000 visitors. This won't take place until July but we have set up a website to chart our progress towards the Exhibit, and this has already had ~300 unique visitors with ~800 article reads.
Year(s) Of Engagement Activity 2017
URL https://medium.com/the-quest-for-a-life-like-prosthetic-hand
 
Description The Future of Surgery 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Policymakers/politicians
Results and Impact Kia Nazarpour has given evidence to the Commission on the Future of Surgery, formed by Royal College of Surgeons. The purpose of the hearing was to set out a compelling and credible vision of the future advances in medicine and technology, including prosthetics, and how those developments will affect the delivery of surgical care in the United Kingdom. In particular, the Commission considers what the future of surgery is likely to look like for patients in five years, could look like in ten years, and might be in 15 to 20 years.
Year(s) Of Engagement Activity 2018
URL https://futureofsurgery.rcseng.ac.uk/report
 
Description Transhumanism 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact This was an interview by The Guardian about the future of Humans and how AI and bionics can create transhumans.
Year(s) Of Engagement Activity 2018
URL https://www.theguardian.com/technology/2018/may/06/no-death-and-an-enhanced-life-is-the-future-trans...
 
Description You have Been Upgraded 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact The team attened Manchester Science Festival to showcase our research in prosthetics control. Members of the public controlled a prosthetic hand with their muscle signals
Year(s) Of Engagement Activity 2018
URL https://www.manchestersciencefestival.com/event/you-have-been-upgraded/