Simultaneous Control of Multiple Degrees of Freedom in Myoelectric Hand Prostheses (SimCon)
Lead Research Organisation:
Newcastle University
Department Name: Electrical, Electronic & Computer Eng
Abstract
The aim of this project is to develop a radically novel and biologically-informed control approach that enables simultaneous control of multiple joints in an upper-limb prosthesis.
The loss of any limb, particularly the hand, affects an individual's quality of life profoundly. Advanced prosthetic hands can provide a route to functional rehabilitation by allowing the amputees to undertake their daily activities and improve their chances of returning to their careers and earning their regular livelihood. Surveys on the use of artificial hands reveal that 20% of the amputees abandon their prosthesis with a key reason being that it does not provide enough function. Therefore, performance enhancement of upper-limb prostheses is a pressing need.
The on-off 1-degree of freedom control paradigm that Reinhold Reiter disclosed in a patent application in 1945 is still used widely for prosthesis control. As early as 1967, Finley showed that the on-off control does not offer enough flexibility to the user and proposed the use of pattern recognition to estimate prosthesis user's movement intention by processing electrical activity of muscles, known as the electromyogram or myoelectric signals. Today, 50 years after Finley's proposal and despite remarkable laboratory demonstrations, it has not been feasible to commercialise pattern recognition in a myoelectric prosthesis hand because 1) it is very difficult for the amputees to generate distinct activity patterns for different movement classes and 2) pattern recognition performance often deteriorates due to electrode displacement and movement of the residual limb.
In this project, we will firstly explore the extent to which muscles in the hand and forearm can learn to generate novel co-contraction patterns (aka muscle synergies) because natural synergies may be disrupted by amputation. The insight gained from this experimental work will inform design of novel algorithms to enable simultaneous control of multiple joints (degrees of freedom) movements. These algorithms can self-tune, to improve performance, as the user interacts with the prosthesis. The project will culminate in a pre-clinical trial in which four amputee subjects test the prototyped control algorithm with a prosthesis. The performance of the proposed paradigm will be compared to that of the conventional prosthesis on-off control method.
The proposed research project will produce an efficient approach for simultaneous control of multiple degrees of freedom. This offers the user much greater flexibility than current on-off or pattern recognition-based control approaches. Pilot results show that with this approach the prosthesis can respond to user's motion intention in only 100ms, which is at least three times faster than the state-of-the-art in upper-limb prostheses control. Results of this research will pave the way for future generations of "plug and play" prostheses with "ready-to-go" and "wearer-independent" features.
The loss of any limb, particularly the hand, affects an individual's quality of life profoundly. Advanced prosthetic hands can provide a route to functional rehabilitation by allowing the amputees to undertake their daily activities and improve their chances of returning to their careers and earning their regular livelihood. Surveys on the use of artificial hands reveal that 20% of the amputees abandon their prosthesis with a key reason being that it does not provide enough function. Therefore, performance enhancement of upper-limb prostheses is a pressing need.
The on-off 1-degree of freedom control paradigm that Reinhold Reiter disclosed in a patent application in 1945 is still used widely for prosthesis control. As early as 1967, Finley showed that the on-off control does not offer enough flexibility to the user and proposed the use of pattern recognition to estimate prosthesis user's movement intention by processing electrical activity of muscles, known as the electromyogram or myoelectric signals. Today, 50 years after Finley's proposal and despite remarkable laboratory demonstrations, it has not been feasible to commercialise pattern recognition in a myoelectric prosthesis hand because 1) it is very difficult for the amputees to generate distinct activity patterns for different movement classes and 2) pattern recognition performance often deteriorates due to electrode displacement and movement of the residual limb.
In this project, we will firstly explore the extent to which muscles in the hand and forearm can learn to generate novel co-contraction patterns (aka muscle synergies) because natural synergies may be disrupted by amputation. The insight gained from this experimental work will inform design of novel algorithms to enable simultaneous control of multiple joints (degrees of freedom) movements. These algorithms can self-tune, to improve performance, as the user interacts with the prosthesis. The project will culminate in a pre-clinical trial in which four amputee subjects test the prototyped control algorithm with a prosthesis. The performance of the proposed paradigm will be compared to that of the conventional prosthesis on-off control method.
The proposed research project will produce an efficient approach for simultaneous control of multiple degrees of freedom. This offers the user much greater flexibility than current on-off or pattern recognition-based control approaches. Pilot results show that with this approach the prosthesis can respond to user's motion intention in only 100ms, which is at least three times faster than the state-of-the-art in upper-limb prostheses control. Results of this research will pave the way for future generations of "plug and play" prostheses with "ready-to-go" and "wearer-independent" features.
Planned Impact
This project paves the way for the development of novel techniques and procedures to realise the first generation of the upper-limb hand prosthesis that offer the possibility of simultaneous control of multiple-degree of freedom. This project will have a huge impact on upper-limb prosthesis users and the society, the economy, knowledge and finally people involved in this project.
Society: Loss of the hand, and the complexities it entails, is one of the most feared conditions having a huge impact upon individuals, their families and society at large. Use of prostheses can improve the quality of life of amputees dramatically and contribute to their personal dignity, independence and more effective inclusion in the society. Advanced prostheses, for instance, those that offer simultaneous multi-joint control, are thus essential to minimise the effect of this devastating condition. In addition to the loss of function that losing a limb causes, phantom pain and psychological distress can be severe. Life-time care can be remarkably expensive. The challenge to provide effective prostheses is a recurrent feature of comments from users. It has been a key theme in national reports on prosthetic services in the UK [Chavasse report, 2014] particularly in relation to injured service personnel and survivors of major trauma. These issues are discussed regularly during meetings of the All-Party Parliamentary Limb Loss Group. Development of efficient and cost-effective upper-limb prosthetic can improve the quality of life of prosthesis users greatly and enable them to return to work.
Economy: The results of this project will cement the UK's reputation as the leading country for prosthetic design and manufacture: 3 of the 4 major manufacturers of upper- and lower-limb prosthetic limbs are UK-based. The market for advanced prostheses and bionics is growing rapidly. For example, Touch Bionics (UK) had a 23% increase in revenue in 2013. In October 2014, it was announced that Touch Bionics will enter the stock market for an estimated value of £50m. This project will provide significant know-how and novel technologies to help the UK's prosthetics industry remain competitive in a global market. Financial security of the British upper-limb prosthetics industry will help to reduce the production and R&D cost allowing the NHS to cover the cost for a larger number of amputees.
Moreover, able-bodied individuals who want advanced interfaces to better operate or interact with robots or games can also benefit from the results of this work. For instance after the release of GetMyo (Thalmic Labs), the use of muscle signals to interact with gadget has become possible for freelance programmers and application developers.
Knowledge: Understanding the flexibility and limits of the motor systems in learning new skills and their application in prosthesis control are very exciting and timely, as evident by the number recent publications in Nature on these topics. As such, the results of this project can lead to a platform to develop the next generation of truly plug and play prosthetic systems.
People: The research will provide high-quality training in the general areas of signal analysis, stochastic learning and sensorimotor control for the research associate (RA1) in this project. RA1 will gain knowledge and expertise through the process of solving challenging problems and presenting his/her work. RA1 will spend time at both Newcastle University's School of Electrical and Electronic Engineering and Institute of Neuroscience and benefit from a truly multi-disciplinary research environment. The Staff Development Unit at Newcastle University offers many training workshops, e.g. on maximising and measuring research impact. RA1 will be advised to attend all relevant workshops. At the end of the project and with such a skills the RA1 will be fully prepared to go to industry or to stay in academia to pursue an academic career.
Society: Loss of the hand, and the complexities it entails, is one of the most feared conditions having a huge impact upon individuals, their families and society at large. Use of prostheses can improve the quality of life of amputees dramatically and contribute to their personal dignity, independence and more effective inclusion in the society. Advanced prostheses, for instance, those that offer simultaneous multi-joint control, are thus essential to minimise the effect of this devastating condition. In addition to the loss of function that losing a limb causes, phantom pain and psychological distress can be severe. Life-time care can be remarkably expensive. The challenge to provide effective prostheses is a recurrent feature of comments from users. It has been a key theme in national reports on prosthetic services in the UK [Chavasse report, 2014] particularly in relation to injured service personnel and survivors of major trauma. These issues are discussed regularly during meetings of the All-Party Parliamentary Limb Loss Group. Development of efficient and cost-effective upper-limb prosthetic can improve the quality of life of prosthesis users greatly and enable them to return to work.
Economy: The results of this project will cement the UK's reputation as the leading country for prosthetic design and manufacture: 3 of the 4 major manufacturers of upper- and lower-limb prosthetic limbs are UK-based. The market for advanced prostheses and bionics is growing rapidly. For example, Touch Bionics (UK) had a 23% increase in revenue in 2013. In October 2014, it was announced that Touch Bionics will enter the stock market for an estimated value of £50m. This project will provide significant know-how and novel technologies to help the UK's prosthetics industry remain competitive in a global market. Financial security of the British upper-limb prosthetics industry will help to reduce the production and R&D cost allowing the NHS to cover the cost for a larger number of amputees.
Moreover, able-bodied individuals who want advanced interfaces to better operate or interact with robots or games can also benefit from the results of this work. For instance after the release of GetMyo (Thalmic Labs), the use of muscle signals to interact with gadget has become possible for freelance programmers and application developers.
Knowledge: Understanding the flexibility and limits of the motor systems in learning new skills and their application in prosthesis control are very exciting and timely, as evident by the number recent publications in Nature on these topics. As such, the results of this project can lead to a platform to develop the next generation of truly plug and play prosthetic systems.
People: The research will provide high-quality training in the general areas of signal analysis, stochastic learning and sensorimotor control for the research associate (RA1) in this project. RA1 will gain knowledge and expertise through the process of solving challenging problems and presenting his/her work. RA1 will spend time at both Newcastle University's School of Electrical and Electronic Engineering and Institute of Neuroscience and benefit from a truly multi-disciplinary research environment. The Staff Development Unit at Newcastle University offers many training workshops, e.g. on maximising and measuring research impact. RA1 will be advised to attend all relevant workshops. At the end of the project and with such a skills the RA1 will be fully prepared to go to industry or to stay in academia to pursue an academic career.
Publications

Abolghasemi V
(2018)
Incoherent Dictionary Pair Learning: Application to a Novel Open-Source Database of Chinese Numbers
in IEEE Signal Processing Letters


Adhikari K
(2016)
A Quaternion Weighted Fourier Linear Combiner for Modeling Physiological Tremor.
in IEEE transactions on bio-medical engineering

Alameer A
(2015)
An elastic net-regularized HMAX model of visual processing

Alameer A
(2020)
Objects and scenes classification with selective use of central and peripheral image content
in Journal of Visual Communication and Image Representation



Alameer A
(2016)
Object Recognition With an Elastic Net-Regularized Hierarchical MAX Model of the Visual Cortex
in IEEE Signal Processing Letters

Title | Hand that Sees Animation |
Description | An animation describing how the hand that sees project works |
Type Of Art | Film/Video/Animation |
Year Produced | 2017 |
Impact | It was used in a major press release and public engagement activity |
URL | http://www.dailymail.co.uk/video/sciencetech/video-1457763/Bionic-hand-sees-objects-decide-best-grip... |
Description | We have shown that the human hand muscles are very flexible in learning new co-activity patterns. This was the first objective of the project. We have also shown that amputees can learn to generate abstract muscle synergies to control a computer interface or a prosthesis. Further studies have now been carried out to show the neural substrate that underpin this flexibility. |
Exploitation Route | We are planning to expand further the work as it is in an early state just yet. The work will tested in volunteer amputee subjects before a wide spread discussion with industry |
Sectors | Electronics,Healthcare |
Description | Impact was in terms of knowledge - the project has led to several publications and some more in preparation. Given the results, we have kicked off a clinical trail with NHS. Over the next years we will be upscaling our work with patients recruited via the NHS. I have given several talks in 2015-2018 about this project and initial results |
First Year Of Impact | 2017 |
Sector | Electronics,Healthcare |
Impact Types | Societal,Economic |
Description | Creation of a User-Centred Vision for Prosthetic Limbs and Clinical Care |
Amount | £250 (GBP) |
Organisation | Port-ER |
Sector | Charity/Non Profit |
Country | United Kingdom |
Start | 03/2018 |
End | 03/2019 |
Description | Healthcare Technology Challenge Award |
Amount | £1,028,683 (GBP) |
Funding ID | EP/R004242/1 |
Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
Sector | Public |
Country | United Kingdom |
Start | 02/2018 |
End | 01/2023 |
Description | Starworks Child Prosthetics |
Amount | £47,790 (GBP) |
Funding ID | STWK-006 |
Organisation | National Institute for Health Research |
Department | NIHR Devices for Dignity Healthcare Technology Co-Operative |
Sector | Public |
Country | United Kingdom |
Start | 03/2018 |
End | 12/2018 |
Description | Systems and Methods for Biofeedback Training |
Amount | £15,078 (GBP) |
Funding ID | KT172076 |
Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
Sector | Public |
Country | United Kingdom |
Start | 07/2017 |
End | 07/2018 |
Title | Myoelectric Controlled Interfaces |
Description | We have developed a computer interface in which a cursor can be controlled by the activity of muscles. The interface can be readily translated to control a hand prosthesis. |
Type Of Material | Physiological assessment or outcome measure |
Year Produced | 2016 |
Provided To Others? | Yes |
Impact | The interface can be used in a variety of application e.g. stroke rehabilitation, in addition to limb prosthetics. We are now finalising the interface to publish it as as open source tool. |
Title | pyEMG |
Description | Python package for offline and real-time myoelectric control by using sEMG and IMU signals |
Type Of Material | Physiological assessment or outcome measure |
Year Produced | 2018 |
Provided To Others? | Yes |
Impact | The impact is growing, as evidenced by new downloads |
URL | https://github.com/agamemnonc/pyEMG |
Title | Newcastle Grasp Library |
Description | 71 objects in four grasp classes are included in this database. We synchronised a Crayfish 55 turntable (Seabass, UK) with a Canon Kiss X4 DSLR camera (resolution 18 Megapixel, 3456×5184 pixels) to take 72 pictures from each object (at 5 degrees intervals) against a black background. |
Type Of Material | Database/Collection of data |
Year Produced | 2017 |
Provided To Others? | Yes |
Impact | This database accompanies a paper published in Journal of Neural Engineering. |
URL | https://rdm.ncl.ac.uk/landing/pages/10.17634/141353-1 |
Title | Online Codebase |
Description | Intelligent Sensing Github |
Type Of Material | Computer model/algorithm |
Year Produced | 2019 |
Provided To Others? | Yes |
Impact | several downloads by others in the field |
URL | https://github.com/intellsensing |
Title | Prosthetcis Control |
Description | We have received approval from NHS to start a small scale clinical trial. We are now in the patient recruiting phase. |
Type | Therapeutic Intervention - Medical Devices |
Current Stage Of Development | Early clinical assessment |
Year Development Stage Completed | 2017 |
Development Status | Under active development/distribution |
Impact | We are now at an early stage. The ideal outcome of the clinical trial would be to be able to predict whether a user can benefit from using an advanced multi-fingered prosthesis or a conventional prosthesis would suffice. This could provide a very much required evidence to support funding for fitting advanced upper limb prosthetics under NHS. |
Description | A talk or presentation - A talk or presentation - Presentation of the work at Trent International Prosthetics Symposium |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Presentation of paper A. Krasoulis, I. Kyranou, M. S. Erden, K. Nazarpour, and S. Vijayakumar, "Improved myoelectric classification via using inertial measurement units," In: ISPO Trent International Prosthetic Symposium (TIPS). 2016, Glasgow: International Society for Prosthetics and Orthotics. |
Year(s) Of Engagement Activity | 2016 |
URL | http://www.ispo.org.uk/events_meetings-V.html |
Description | A talk or presentation - Presentation of the work at Trent International Prosthetics Symposium |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Presentation of research work at Trent International Prosthetics Symposium (combined with the International Society for Prosthetics and Orthotics), a multidisciplinary conference attracting not just academics but industry, users and practitioners. M. Dyson and K. Nazarpour "Pre-clinical application of abstract muscle synergies for myoelectric control," In: ISPO Trent International Prosthetic Symposium (TIPS). 2016, Glasgow: International Society for Prosthetics and Orthotics |
Year(s) Of Engagement Activity | 2016 |
URL | http://www.ispo.org.uk/events_meetings-V.html |
Description | A talk at the BioNow conference (Newcastle) |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Regional |
Primary Audience | Industry/Business |
Results and Impact | An event to bring together medical engineering research at Newcastle University and medical and biotechnology industry in North East England. |
Year(s) Of Engagement Activity | Pre-2006,2016 |
URL | https://twitter.com/search?q=%23biofocus |
Description | A talk or presentation - Imperial College Surgical Society |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Undergraduate students |
Results and Impact | A talk about body integrated prosthetic devices, surgical challenges and skills required For the link, please refer to the tweets on Jan 15 |
Year(s) Of Engagement Activity | 2017 |
URL | https://twitter.com/ICSMSurgicalSoc |
Description | A talk presented at Sheffiled University |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | The CATCH Network learned about machine learning in the control of prosthetic hands and Patient and Public Involvement. We were delighted to be joined by not one, not two, but three exciting speakers at the June 2016 CATCH Networking Lunch. |
Year(s) Of Engagement Activity | 2017 |
URL | http://www.catch.org.uk/news-articles/6th-networking-lunch/ |
Description | Future prosthetic: towards the bionic human |
Form Of Engagement Activity | A magazine, newsletter or online publication |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | Kia Nazarpour gave an interview to The Engineer Magazine about the science-fiction vision of robotic prosthetic limbs that can be controlled by the brain and provide sensory feedback, which in his opinion is coming closer to reality. |
Year(s) Of Engagement Activity | 2018 |
URL | https://www.theengineer.co.uk/future-prosthetic/ |
Description | International Press Release |
Form Of Engagement Activity | A press release, press conference or response to a media enquiry/interview |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Schools |
Results and Impact | Press Release on the Hand that Sees Project |
Year(s) Of Engagement Activity | 2017 |
URL | http://www.bbc.co.uk/news/uk-england-tyne-39797011 |
Description | International Press Release |
Form Of Engagement Activity | A press release, press conference or response to a media enquiry/interview |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Media (as a channel to the public) |
Results and Impact | Press release about the UNESCO award received by the hand that sees project |
Year(s) Of Engagement Activity | 2018 |
URL | http://www.ncl.ac.uk/press/articles/latest/2018/02/handthatseesunesco/ |
Description | Interview |
Form Of Engagement Activity | A magazine, newsletter or online publication |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | Gave an interview to The Engineer on the science-fiction vision of robotic prosthetic limbs that can be controlled by the brain and provide sensory feedback is coming closer. |
Year(s) Of Engagement Activity | 2018 |
URL | https://www.theengineer.co.uk/future-prosthetic/ |
Description | Interview with Dazed |
Form Of Engagement Activity | A press release, press conference or response to a media enquiry/interview |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | Interview about "How AI could increase art world accessibility for disabled artists" |
Year(s) Of Engagement Activity | 2018 |
URL | https://www.dazeddigital.com/art-photography/article/41334/1/how-ai-could-increase-art-world-accessi... |
Description | Interview with Verdict |
Form Of Engagement Activity | A magazine, newsletter or online publication |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | This was an interview about how AI is exploding into healthcare |
Year(s) Of Engagement Activity | 2018 |
URL | https://www.verdict.co.uk/ai-healthcare-growth-apps/ |
Description | Newcastle High School for Girls Loves STEM event - A School Visit |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Schools |
Results and Impact | Showcasing bionics research in an STEM event in a local girls only school |
Year(s) Of Engagement Activity | 2017 |
Description | School Visit |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Schools |
Results and Impact | Presenting control of myoelectric hands in a local girls only school |
Year(s) Of Engagement Activity | 2017 |
URL | http://www.intellsensing.com/outreach-we-love-stem/ |
Description | Speaking at IET Annual Lecture |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Postgraduate students |
Results and Impact | Presented challenges of myoelectric control in a public event |
Year(s) Of Engagement Activity | 2017 |
URL | http://www.theiet.org/events/local/250075.cfm |
Description | The Future of Surgery |
Form Of Engagement Activity | A formal working group, expert panel or dialogue |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Policymakers/politicians |
Results and Impact | Kia Nazarpour has given evidence to the Commission on the Future of Surgery, formed by Royal College of Surgeons. The purpose of the hearing was to set out a compelling and credible vision of the future advances in medicine and technology, including prosthetics, and how those developments will affect the delivery of surgical care in the United Kingdom. In particular, the Commission considers what the future of surgery is likely to look like for patients in five years, could look like in ten years, and might be in 15 to 20 years. |
Year(s) Of Engagement Activity | 2018 |
URL | https://futureofsurgery.rcseng.ac.uk/report |
Description | Transhumanism |
Form Of Engagement Activity | A magazine, newsletter or online publication |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | This was an interview by The Guardian about the future of Humans and how AI and bionics can create transhumans. |
Year(s) Of Engagement Activity | 2018 |
URL | https://www.theguardian.com/technology/2018/may/06/no-death-and-an-enhanced-life-is-the-future-trans... |
Description | You have Been Upgraded |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | Regional |
Primary Audience | Public/other audiences |
Results and Impact | The team attened Manchester Science Festival to showcase our research in prosthetics control. Members of the public controlled a prosthetic hand with their muscle signals |
Year(s) Of Engagement Activity | 2018 |
URL | https://www.manchestersciencefestival.com/event/you-have-been-upgraded/ |