Sensorimotor Learning for Control of Prosthetic Limbs
Lead Research Organisation:
Newcastle University
Department Name: Sch of Engineering
Abstract
Worldwide, there are over three million people living with upper-limb loss. Recent wars, industrialisation in developing countries and vascular disease, e.g. diabetes, have caused the number of amputations to soar. Adding to this population each year, one in every 2,500 people are born with upper-limb reduction. Advanced prostheses can play a major role in enhancing the quality of life for people with upper-limb loss, however, they are not available under the NHS. Notably, many people with traumatic limb loss are otherwise physically fit. If they are equipped with advanced prostheses and treated to recover psychologically, they can live independently, with minimal need for social support, return to work and contribute to the economy.
There are a plethora of underlying reasons that limit wide clinical adoption of advanced prosthetic hands. For instance, surveys on their use reveal that 20% of upper-limb amputees abandon their prosthesis, with the primary reason being that the control of these systems is still limited to one or two movements. In addition, the process of switching a prosthetic hand into an appropriate grip mode, e.g. to use scissors, is cumbersome or requires an ad-hoc solution, such as using a smart phone application. Other reasons include: users finding their prosthesis uncomfortable or unsuitable for their needs. As such, everyday tasks, such as tying shoe-laces, are currently very challenging for prosthetic hand users. These functional shortcomings, coupled with high costs and lack of concrete evidence for added benefit, have emerged as substantial barriers limiting clinical adoption of advanced prosthetic hands.
The long-term aim of this cross-disciplinary programme is to develop, and move towards making available, the next generation of prosthetic hands that can improve the users' quality of life. Our underlying scientific novelty is in utilising users' capability of learning to operate a prosthesis. For instance, we examine the extent to which the activity of muscles can deviate from natural patterns employed in controlling movement of the biological arm and hand and whether prosthesis users can learn to synthesise these functional maps between muscles and prosthetic digits. Basing this approach upon our pilot data, we hypothesise that practice and availability of sensory feedback can accelerate this learning experience. To address this fundamental question, we will employ in-vivo experiments, exploratory studies involving able-bodied volunteers and pre-clinical work with people with limb loss. The insight gained from these studies will inform the design of novel algorithms to enable seamless control of prosthetic hands. Finally, the programme will culminate with a unifying theory for learning to control prosthetic hands that will be tested in an NHS-approved, pre-clinical trial.
Maturing this approach into a clinically-viable solution needs a dedicated team of engineers and scientists as well as a consortium of users, NHS-based clinicians and healthcare and high-tech industries. With the flexibility that a Healthcare Technologies Challenge Award affords me, I will be able to nurture and grow sustainably my multi-disciplinary team. In addition, this flexible funding will enable to focus on a converging research programme with the ultimate aim of providing prosthetic solutions that enhance NHS-approved clinical patient outcome measures significantly.
Within this programme, I will identify and bring together the engineering, scientific, clinical, ethical and regulatory elements necessary to form a recognised national hub for the development of next-generation prosthetics. This work will provide the foundations for my 15-year plan to establish the Centre for Bionic Limbs. The origin of this Centre will be to act as a mechanism to safeguard engineering and scientific innovations, increase value, and accelerate transfer into commercial and clinical fields.
There are a plethora of underlying reasons that limit wide clinical adoption of advanced prosthetic hands. For instance, surveys on their use reveal that 20% of upper-limb amputees abandon their prosthesis, with the primary reason being that the control of these systems is still limited to one or two movements. In addition, the process of switching a prosthetic hand into an appropriate grip mode, e.g. to use scissors, is cumbersome or requires an ad-hoc solution, such as using a smart phone application. Other reasons include: users finding their prosthesis uncomfortable or unsuitable for their needs. As such, everyday tasks, such as tying shoe-laces, are currently very challenging for prosthetic hand users. These functional shortcomings, coupled with high costs and lack of concrete evidence for added benefit, have emerged as substantial barriers limiting clinical adoption of advanced prosthetic hands.
The long-term aim of this cross-disciplinary programme is to develop, and move towards making available, the next generation of prosthetic hands that can improve the users' quality of life. Our underlying scientific novelty is in utilising users' capability of learning to operate a prosthesis. For instance, we examine the extent to which the activity of muscles can deviate from natural patterns employed in controlling movement of the biological arm and hand and whether prosthesis users can learn to synthesise these functional maps between muscles and prosthetic digits. Basing this approach upon our pilot data, we hypothesise that practice and availability of sensory feedback can accelerate this learning experience. To address this fundamental question, we will employ in-vivo experiments, exploratory studies involving able-bodied volunteers and pre-clinical work with people with limb loss. The insight gained from these studies will inform the design of novel algorithms to enable seamless control of prosthetic hands. Finally, the programme will culminate with a unifying theory for learning to control prosthetic hands that will be tested in an NHS-approved, pre-clinical trial.
Maturing this approach into a clinically-viable solution needs a dedicated team of engineers and scientists as well as a consortium of users, NHS-based clinicians and healthcare and high-tech industries. With the flexibility that a Healthcare Technologies Challenge Award affords me, I will be able to nurture and grow sustainably my multi-disciplinary team. In addition, this flexible funding will enable to focus on a converging research programme with the ultimate aim of providing prosthetic solutions that enhance NHS-approved clinical patient outcome measures significantly.
Within this programme, I will identify and bring together the engineering, scientific, clinical, ethical and regulatory elements necessary to form a recognised national hub for the development of next-generation prosthetics. This work will provide the foundations for my 15-year plan to establish the Centre for Bionic Limbs. The origin of this Centre will be to act as a mechanism to safeguard engineering and scientific innovations, increase value, and accelerate transfer into commercial and clinical fields.
Planned Impact
This programme aims at development of a new biologically-informed approach, termed Pattern Learning, for the control of upper-limb hand prostheses. The impact of this programme is as the following:
Society: Loss of the hand, and the complexities it entails, is one of the most feared conditions having a large impact upon individuals, their families and the society. In addition to the loss of function that losing a limb causes, phantom pain and psychological distress can be severe. Life-time care requires taking drugs that have side effects. Use of advanced prostheses can improve the quality of life of users dramatically and contribute to their personal dignity, independence and more effective inclusion in the society. As such, prostheses that offer seamless control of multi-joint movements are essential to minimise the effect of limb loss.
The challenge to provide effective prostheses is a recurrent feature of comments from users in Patient and Public Involvement meetings. In addition, It has been a key theme in recent national reports on prosthetic services in the UK (e.g. Chavasse report, 2014) particularly in relation to injured service personnel and survivors of major trauma. These issues are discussed regularly during meetings of the All-Party Parliamentary Limb Loss Group. The proposed programme has the potential to make a step change in the quality of control of prosthetic hands.
In addition to prosthetics, the knowledge and technologies, e.g. stimulation of the peripheral nerves, that we will develop in this programme can be translated to other movement disorders such as foot drop, a muscular weakness or paralysis that makes walking difficult because the patient cannot lift the front part of the foot and toes.
Economy: The results of this project will cement the UK's reputation as the leading country for prosthetic design and manufacturing. Three of the four major manufacturers of upper- and lower-limb prosthetic limbs are based in the UK. The market for advanced prostheses and bionics is growing rapidly. This project will provide significant know-how and novel technologies to help the UK's prosthetics industry remain competitive in a global market. Financial security of the British upper-limb prosthetics industry will help to reduce the production and R&D cost allowing the NHS to cover the cost for a larger number of patients. Moreover, able-bodied individuals who want advanced interfaces to better operate or interact with robots or games can also benefit from the results of this work. In addition, the implantable electronic technologies for neural stimulation that we develop can be taken up by the Electronic Medicine (electroceuticals) industry and used for treatment of a range of health conditions such as diabetes and depression.
Knowledge: Understanding the flexibility and limits of the human motor systems in learning new skills and their application in prosthesis control are very exciting and timely, as evident by the number recent publications in Nature on these topics. Our cross-species approach will help development of a unifying theory for Pattern Learning which is very exciting scientifically because with Pattern Learning we will be able to develop a platform for truly plug and play prosthetic systems.
People: This programme will provide me with a platform upon which I can work towards becoming a research leader in the field of prosthetics. In addition, it will provide high-quality training in the general areas of signal analysis and sensorimotor control for three research associates (RA1-3) and several PhD students. All team members will gain knowledge and expertise through the process of solving challenging problems. They will spend time at both Newcastle University's School of Electrical and Electronic Engineering and Institute of Neuroscience as well as at collaborating hospitals (Freeman and Salford Royal). They will benefit from a truly multi-disciplinary research environment.
Society: Loss of the hand, and the complexities it entails, is one of the most feared conditions having a large impact upon individuals, their families and the society. In addition to the loss of function that losing a limb causes, phantom pain and psychological distress can be severe. Life-time care requires taking drugs that have side effects. Use of advanced prostheses can improve the quality of life of users dramatically and contribute to their personal dignity, independence and more effective inclusion in the society. As such, prostheses that offer seamless control of multi-joint movements are essential to minimise the effect of limb loss.
The challenge to provide effective prostheses is a recurrent feature of comments from users in Patient and Public Involvement meetings. In addition, It has been a key theme in recent national reports on prosthetic services in the UK (e.g. Chavasse report, 2014) particularly in relation to injured service personnel and survivors of major trauma. These issues are discussed regularly during meetings of the All-Party Parliamentary Limb Loss Group. The proposed programme has the potential to make a step change in the quality of control of prosthetic hands.
In addition to prosthetics, the knowledge and technologies, e.g. stimulation of the peripheral nerves, that we will develop in this programme can be translated to other movement disorders such as foot drop, a muscular weakness or paralysis that makes walking difficult because the patient cannot lift the front part of the foot and toes.
Economy: The results of this project will cement the UK's reputation as the leading country for prosthetic design and manufacturing. Three of the four major manufacturers of upper- and lower-limb prosthetic limbs are based in the UK. The market for advanced prostheses and bionics is growing rapidly. This project will provide significant know-how and novel technologies to help the UK's prosthetics industry remain competitive in a global market. Financial security of the British upper-limb prosthetics industry will help to reduce the production and R&D cost allowing the NHS to cover the cost for a larger number of patients. Moreover, able-bodied individuals who want advanced interfaces to better operate or interact with robots or games can also benefit from the results of this work. In addition, the implantable electronic technologies for neural stimulation that we develop can be taken up by the Electronic Medicine (electroceuticals) industry and used for treatment of a range of health conditions such as diabetes and depression.
Knowledge: Understanding the flexibility and limits of the human motor systems in learning new skills and their application in prosthesis control are very exciting and timely, as evident by the number recent publications in Nature on these topics. Our cross-species approach will help development of a unifying theory for Pattern Learning which is very exciting scientifically because with Pattern Learning we will be able to develop a platform for truly plug and play prosthetic systems.
People: This programme will provide me with a platform upon which I can work towards becoming a research leader in the field of prosthetics. In addition, it will provide high-quality training in the general areas of signal analysis and sensorimotor control for three research associates (RA1-3) and several PhD students. All team members will gain knowledge and expertise through the process of solving challenging problems. They will spend time at both Newcastle University's School of Electrical and Electronic Engineering and Institute of Neuroscience as well as at collaborating hospitals (Freeman and Salford Royal). They will benefit from a truly multi-disciplinary research environment.
Organisations
Publications
Abolghasemi V
(2018)
Incoherent Dictionary Pair Learning: Application to a Novel Open-Source Database of Chinese Numbers
in IEEE Signal Processing Letters
Alameer A
(2020)
Objects and scenes classification with selective use of central and peripheral image content
in Journal of Visual Communication and Image Representation
Blana D
(2020)
Model-Based Control of Individual Finger Movements for Prosthetic Hand Function.
in IEEE transactions on neural systems and rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society
Brunton E
(2018)
Recovery of the Response of Sensory Fibers to the Second of a Pair of Peripheral Nerve Stimuli.
in Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual International Conference
Brunton EK
(2019)
Temporal Modulation of the Response of Sensory Fibers to Paired-Pulse Stimulation.
in IEEE transactions on neural systems and rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society
Dyson M
(2018)
Data Driven Spatial Filtering Can Enhance Abstract Myoelectric Control in Amputees.
in Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual International Conference
Dyson M
(2020)
Learning, Generalization, and Scalability of Abstract Myoelectric Control.
in IEEE transactions on neural systems and rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society
Dyson M
(2018)
Abstract Decoding using Bayesian Muscle Activation Estimators.
in Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual International Conference
Title | Supplementary video from IoT for beyond-the-laboratory prosthetics research |
Description | Video clips for online data collection, device monitoring and user feedback submission during the pick and place experiment |
Type Of Art | Film/Video/Animation |
Year Produced | 2022 |
URL | https://rs.figshare.com/articles/media/Supplementary_video_from_IoT_for_beyond-the-laboratory_prosth... |
Title | Supplementary video from IoT for beyond-the-laboratory prosthetics research |
Description | Video clips for online data collection, device monitoring and user feedback submission during the pick and place experiment |
Type Of Art | Film/Video/Animation |
Year Produced | 2022 |
URL | https://rs.figshare.com/articles/media/Supplementary_video_from_IoT_for_beyond-the-laboratory_prosth... |
Description | When controlling a prosthesis, the patterns of muscular activity can differ from those used to control the biological limbs. |
Exploitation Route | 2019 The above finding can radically change the way we look at controlling prosthetic limbs. We are at an early stage of collaborating with a company to embed this approach in their controller. |
Sectors | Electronics Healthcare |
Description | 2019 This is a 5-year programme. The main impact in Year 1 has been on the engagement of the public with STEM and prosthetics in particular |
First Year Of Impact | 2018 |
Sector | Electronics,Healthcare |
Impact Types | Societal |
Description | Animal Free UK |
Amount | £2,000 (GBP) |
Organisation | Animal Free Research UK |
Sector | Charity/Non Profit |
Country | United Kingdom |
Start | 05/2019 |
End | 08/2019 |
Description | Creation of a User-Centred Vision for Prosthetic Limbs and Clinical Care |
Amount | £250 (GBP) |
Organisation | Port-ER |
Sector | Charity/Non Profit |
Country | United Kingdom |
Start | 03/2018 |
End | 03/2019 |
Description | RISE-WELL Critical solutions for elderly well-being |
Amount | € 2,691,279 (EUR) |
Funding ID | DLV-860173 |
Organisation | European Commission H2020 |
Sector | Public |
Country | Belgium |
Start | 09/2019 |
End | 09/2023 |
Description | Starworks Child Prosthetics |
Amount | £47,790 (GBP) |
Funding ID | STWK-006 |
Organisation | National Institute for Health Research |
Department | NIHR Devices for Dignity Healthcare Technology Co-Operative |
Sector | Public |
Country | United Kingdom |
Start | 03/2018 |
End | 12/2018 |
Description | UKIERI |
Amount | £12,000 (GBP) |
Organisation | UK-India Education and Research Initiative (UKIERI) |
Sector | Academic/University |
Country | United Kingdom |
Start | 08/2019 |
End | 08/2021 |
Title | pyEMG |
Description | Python package for offline and real-time myoelectric control by using sEMG and IMU signals |
Type Of Material | Physiological assessment or outcome measure |
Year Produced | 2018 |
Provided To Others? | Yes |
Impact | The impact is growing, as evidenced by new downloads |
URL | https://github.com/agamemnonc/pyEMG |
Title | Data for 'IoT for beyond-the-laboratory prosthetics research' |
Description | The original data for the pick and place experiment and the MATLAB script to plot the data |
Type Of Material | Database/Collection of data |
Year Produced | 2022 |
Provided To Others? | Yes |
URL | https://rs.figshare.com/articles/dataset/Data_for_IoT_for_beyond-the-laboratory_prosthetics_research... |
Title | Data for 'IoT for beyond-the-laboratory prosthetics research' |
Description | The original data for the pick and place experiment and the MATLAB script to plot the data |
Type Of Material | Database/Collection of data |
Year Produced | 2022 |
Provided To Others? | Yes |
URL | https://rs.figshare.com/articles/dataset/Data_for_IoT_for_beyond-the-laboratory_prosthetics_research... |
Title | ENG recordings in response to mechanical stimulation |
Description | Electroneurogram signals recorded from 16-channel cuff electrodes placed on the sciatic nerve of Sprague Dawley rats. The recordings were made in response to three types of mechanical stimulation, namely, proprioception, touch and nociception. The data files include synchronisation comments and signals that indicate when a stimulus was applied and what the stimulus was. Proprioceptive responses were recorded in response to a servo motor moving the foot to one of 6 different angles. Touch responses were recorded in response to heel of the foot being touched with one of two Von Frey fibres, corresponding to a force of either 100 or 300 grams. Finally, Nociception responses were recorded in response to the foot being pinched using a pair of forceps either on the heel, or the outer toe. |
Type Of Material | Database/Collection of data |
Year Produced | 2017 |
Provided To Others? | Yes |
Impact | Experiments with animals create large amount valuable data on which various algorithms can be applied. This is a first database of its kind which made available online. We hope it will help with the reduction of number of animals used in research. |
URL | https://rdm.ncl.ac.uk/landing/pages/10.17634/141353-2 |
Title | Online Codebase |
Description | Intelligent Sensing Github |
Type Of Material | Computer model/algorithm |
Year Produced | 2019 |
Provided To Others? | Yes |
Impact | several downloads by others in the field |
URL | https://github.com/intellsensing |
Company Name | Neuranics |
Description | Neuranics develops magnetic neurotechnology intended to integrate with wearables and implantable devices. |
Year Established | 2021 |
Impact | Early stage but so far the company created 11 jobs |
Website | https://neuranics.com/ |
Description | A patient public involvement event |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Other audiences |
Results and Impact | In December 2018, the Intelligent Sensing laboratory at Newcastle University hosted a one-day workshop to capture challenges and opportunities to improve the lives of people living with limb loss. This was a collaborative project that aimed to create a user-centred vision for prosthetic devices and clinical care. In order to achieve this aim, the project team required interaction with a variety of different stakeholder. Therefore the workshop was developed to benefit academics, NHS clinicians, industry specialists¬, charity executives and prosthesis users; all of which were in attendance at the workshop. |
Year(s) Of Engagement Activity | 2018 |
Description | A talk presented at Sheffiled University |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | The CATCH Network learned about machine learning in the control of prosthetic hands and Patient and Public Involvement. We were delighted to be joined by not one, not two, but three exciting speakers at the June 2016 CATCH Networking Lunch. |
Year(s) Of Engagement Activity | 2017 |
URL | http://www.catch.org.uk/news-articles/6th-networking-lunch/ |
Description | Future prosthetic: towards the bionic human |
Form Of Engagement Activity | A magazine, newsletter or online publication |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | Kia Nazarpour gave an interview to The Engineer Magazine about the science-fiction vision of robotic prosthetic limbs that can be controlled by the brain and provide sensory feedback, which in his opinion is coming closer to reality. |
Year(s) Of Engagement Activity | 2018 |
URL | https://www.theengineer.co.uk/future-prosthetic/ |
Description | Interview with Dazed |
Form Of Engagement Activity | A press release, press conference or response to a media enquiry/interview |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | Interview about "How AI could increase art world accessibility for disabled artists" |
Year(s) Of Engagement Activity | 2018 |
URL | https://www.dazeddigital.com/art-photography/article/41334/1/how-ai-could-increase-art-world-accessi... |
Description | Interview with Verdict |
Form Of Engagement Activity | A magazine, newsletter or online publication |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | This was an interview about how AI is exploding into healthcare |
Year(s) Of Engagement Activity | 2018 |
URL | https://www.verdict.co.uk/ai-healthcare-growth-apps/ |
Description | Royal Society Exhibition Blog |
Form Of Engagement Activity | Engagement focused website, blog or social media channel |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | We have been selected by the Royal Society to exhibit our work at their Summer Science FEstival, attracting around 14000 visitors. This won't take place until July but we have set up a website to chart our progress towards the Exhibit, and this has already had ~300 unique visitors with ~800 article reads. |
Year(s) Of Engagement Activity | 2017 |
URL | https://medium.com/the-quest-for-a-life-like-prosthetic-hand |
Description | The Future of Surgery |
Form Of Engagement Activity | A formal working group, expert panel or dialogue |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Policymakers/politicians |
Results and Impact | Kia Nazarpour has given evidence to the Commission on the Future of Surgery, formed by Royal College of Surgeons. The purpose of the hearing was to set out a compelling and credible vision of the future advances in medicine and technology, including prosthetics, and how those developments will affect the delivery of surgical care in the United Kingdom. In particular, the Commission considers what the future of surgery is likely to look like for patients in five years, could look like in ten years, and might be in 15 to 20 years. |
Year(s) Of Engagement Activity | 2018 |
URL | https://futureofsurgery.rcseng.ac.uk/report |
Description | Transhumanism |
Form Of Engagement Activity | A magazine, newsletter or online publication |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | This was an interview by The Guardian about the future of Humans and how AI and bionics can create transhumans. |
Year(s) Of Engagement Activity | 2018 |
URL | https://www.theguardian.com/technology/2018/may/06/no-death-and-an-enhanced-life-is-the-future-trans... |
Description | You have Been Upgraded |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | Regional |
Primary Audience | Public/other audiences |
Results and Impact | The team attened Manchester Science Festival to showcase our research in prosthetics control. Members of the public controlled a prosthetic hand with their muscle signals |
Year(s) Of Engagement Activity | 2018 |
URL | https://www.manchestersciencefestival.com/event/you-have-been-upgraded/ |