Simultaneous Control of Multiple Degrees of Freedom in Myoelectric Hand Prostheses (SimCon)

Lead Research Organisation: Newcastle University
Department Name: Electrical, Electronic & Computer Eng

Abstract

The aim of this project is to develop a radically novel and biologically-informed control approach that enables simultaneous control of multiple joints in an upper-limb prosthesis.

The loss of any limb, particularly the hand, affects an individual's quality of life profoundly. Advanced prosthetic hands can provide a route to functional rehabilitation by allowing the amputees to undertake their daily activities and improve their chances of returning to their careers and earning their regular livelihood. Surveys on the use of artificial hands reveal that 20% of the amputees abandon their prosthesis with a key reason being that it does not provide enough function. Therefore, performance enhancement of upper-limb prostheses is a pressing need.

The on-off 1-degree of freedom control paradigm that Reinhold Reiter disclosed in a patent application in 1945 is still used widely for prosthesis control. As early as 1967, Finley showed that the on-off control does not offer enough flexibility to the user and proposed the use of pattern recognition to estimate prosthesis user's movement intention by processing electrical activity of muscles, known as the electromyogram or myoelectric signals. Today, 50 years after Finley's proposal and despite remarkable laboratory demonstrations, it has not been feasible to commercialise pattern recognition in a myoelectric prosthesis hand because 1) it is very difficult for the amputees to generate distinct activity patterns for different movement classes and 2) pattern recognition performance often deteriorates due to electrode displacement and movement of the residual limb.

In this project, we will firstly explore the extent to which muscles in the hand and forearm can learn to generate novel co-contraction patterns (aka muscle synergies) because natural synergies may be disrupted by amputation. The insight gained from this experimental work will inform design of novel algorithms to enable simultaneous control of multiple joints (degrees of freedom) movements. These algorithms can self-tune, to improve performance, as the user interacts with the prosthesis. The project will culminate in a pre-clinical trial in which four amputee subjects test the prototyped control algorithm with a prosthesis. The performance of the proposed paradigm will be compared to that of the conventional prosthesis on-off control method.

The proposed research project will produce an efficient approach for simultaneous control of multiple degrees of freedom. This offers the user much greater flexibility than current on-off or pattern recognition-based control approaches. Pilot results show that with this approach the prosthesis can respond to user's motion intention in only 100ms, which is at least three times faster than the state-of-the-art in upper-limb prostheses control. Results of this research will pave the way for future generations of "plug and play" prostheses with "ready-to-go" and "wearer-independent" features.

Planned Impact

This project paves the way for the development of novel techniques and procedures to realise the first generation of the upper-limb hand prosthesis that offer the possibility of simultaneous control of multiple-degree of freedom. This project will have a huge impact on upper-limb prosthesis users and the society, the economy, knowledge and finally people involved in this project.

Society: Loss of the hand, and the complexities it entails, is one of the most feared conditions having a huge impact upon individuals, their families and society at large. Use of prostheses can improve the quality of life of amputees dramatically and contribute to their personal dignity, independence and more effective inclusion in the society. Advanced prostheses, for instance, those that offer simultaneous multi-joint control, are thus essential to minimise the effect of this devastating condition. In addition to the loss of function that losing a limb causes, phantom pain and psychological distress can be severe. Life-time care can be remarkably expensive. The challenge to provide effective prostheses is a recurrent feature of comments from users. It has been a key theme in national reports on prosthetic services in the UK [Chavasse report, 2014] particularly in relation to injured service personnel and survivors of major trauma. These issues are discussed regularly during meetings of the All-Party Parliamentary Limb Loss Group. Development of efficient and cost-effective upper-limb prosthetic can improve the quality of life of prosthesis users greatly and enable them to return to work.

Economy: The results of this project will cement the UK's reputation as the leading country for prosthetic design and manufacture: 3 of the 4 major manufacturers of upper- and lower-limb prosthetic limbs are UK-based. The market for advanced prostheses and bionics is growing rapidly. For example, Touch Bionics (UK) had a 23% increase in revenue in 2013. In October 2014, it was announced that Touch Bionics will enter the stock market for an estimated value of £50m. This project will provide significant know-how and novel technologies to help the UK's prosthetics industry remain competitive in a global market. Financial security of the British upper-limb prosthetics industry will help to reduce the production and R&D cost allowing the NHS to cover the cost for a larger number of amputees.

Moreover, able-bodied individuals who want advanced interfaces to better operate or interact with robots or games can also benefit from the results of this work. For instance after the release of GetMyo (Thalmic Labs), the use of muscle signals to interact with gadget has become possible for freelance programmers and application developers.

Knowledge: Understanding the flexibility and limits of the motor systems in learning new skills and their application in prosthesis control are very exciting and timely, as evident by the number recent publications in Nature on these topics. As such, the results of this project can lead to a platform to develop the next generation of truly plug and play prosthetic systems.

People: The research will provide high-quality training in the general areas of signal analysis, stochastic learning and sensorimotor control for the research associate (RA1) in this project. RA1 will gain knowledge and expertise through the process of solving challenging problems and presenting his/her work. RA1 will spend time at both Newcastle University's School of Electrical and Electronic Engineering and Institute of Neuroscience and benefit from a truly multi-disciplinary research environment. The Staff Development Unit at Newcastle University offers many training workshops, e.g. on maximising and measuring research impact. RA1 will be advised to attend all relevant workshops. At the end of the project and with such a skills the RA1 will be fully prepared to go to industry or to stay in academia to pursue an academic career.
 
Title Hand that Sees Animation 
Description An animation describing how the hand that sees project works 
Type Of Art Film/Video/Animation 
Year Produced 2017 
Impact It was used in a major press release and public engagement activity 
URL http://www.dailymail.co.uk/video/sciencetech/video-1457763/Bionic-hand-sees-objects-decide-best-grip...
 
Description We have shown that the human hand muscles are very flexible in learning new co-activity patterns. This was the first objective of the project. We have also shown that amputees can learn to generate abstract muscle synergies to control a computer interface or a prosthesis. Further studies have now been carried out to show the neural substrate that underpin this flexibility.
Exploitation Route We are planning to expand further the work as it is in an early state just yet. The work will tested in volunteer amputee subjects before a wide spread discussion with industry
Sectors Electronics,Healthcare

 
Description Impact was in terms of knowledge - the project has led to several publications and some more in preparation. Given the results, we have kicked off a clinical trail with NHS. Over the next years we will be upscaling our work with patients recruited via the NHS. I have given several talks in 2015-2018 about this project and initial results
First Year Of Impact 2017
Sector Electronics,Healthcare
Impact Types Societal,Economic

 
Description Healthcare Technology Challenge Award
Amount £1,028,683 (GBP)
Funding ID EP/R004242/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Academic/University
Country United Kingdom
Start 02/2018 
End 01/2023
 
Description Starworks Child Prosthetics
Amount £47,790 (GBP)
Funding ID STWK-006 
Organisation National Institute for Health Research 
Department NIHR Devices for Dignity Healthcare Technology Co-Operative
Sector Public
Country United Kingdom
Start 03/2018 
End 12/2018
 
Description Systems and Methods for Biofeedback Training
Amount £15,078 (GBP)
Funding ID KT172076 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Academic/University
Country United Kingdom
Start 08/2017 
End 07/2018
 
Title Myoelectric Controlled Interfaces 
Description We have developed a computer interface in which a cursor can be controlled by the activity of muscles. The interface can be readily translated to control a hand prosthesis. 
Type Of Material Physiological assessment or outcome measure 
Year Produced 2016 
Provided To Others? Yes  
Impact The interface can be used in a variety of application e.g. stroke rehabilitation, in addition to limb prosthetics. We are now finalising the interface to publish it as as open source tool. 
 
Title pyEMG 
Description Python package for offline and real-time myoelectric control by using sEMG and IMU signals 
Type Of Material Physiological assessment or outcome measure 
Year Produced 2018 
Provided To Others? Yes  
Impact The impact is growing, as evidenced by new downloads 
URL https://github.com/agamemnonc/pyEMG
 
Title Newcastle Grasp Library 
Description 71 objects in four grasp classes are included in this database. We synchronised a Crayfish 55 turntable (Seabass, UK) with a Canon Kiss X4 DSLR camera (resolution 18 Megapixel, 3456×5184 pixels) to take 72 pictures from each object (at 5 degrees intervals) against a black background. 
Type Of Material Database/Collection of data 
Year Produced 2017 
Provided To Others? Yes  
Impact This database accompanies a paper published in Journal of Neural Engineering. 
URL https://rdm.ncl.ac.uk/landing/pages/10.17634/141353-1
 
Title Prosthetcis Control 
Description We have received approval from NHS to start a small scale clinical trial. We are now in the patient recruiting phase. 
Type Therapeutic Intervention - Medical Devices
Current Stage Of Development Early clinical assessment
Year Development Stage Completed 2017
Development Status Under active development/distribution
Impact We are now at an early stage. The ideal outcome of the clinical trial would be to be able to predict whether a user can benefit from using an advanced multi-fingered prosthesis or a conventional prosthesis would suffice. This could provide a very much required evidence to support funding for fitting advanced upper limb prosthetics under NHS. 
 
Description A talk or presentation - A talk or presentation - Presentation of the work at Trent International Prosthetics Symposium 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Presentation of paper
A. Krasoulis, I. Kyranou, M. S. Erden, K. Nazarpour, and S. Vijayakumar, "Improved myoelectric classification via using inertial measurement units," In: ISPO Trent International Prosthetic Symposium (TIPS). 2016, Glasgow: International Society for Prosthetics and Orthotics.
Year(s) Of Engagement Activity 2016
URL http://www.ispo.org.uk/events_meetings-V.html
 
Description A talk or presentation - Presentation of the work at Trent International Prosthetics Symposium 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Presentation of research work at Trent International Prosthetics Symposium (combined with the International Society for Prosthetics and Orthotics), a multidisciplinary conference attracting not just academics but industry, users and practitioners.

M. Dyson and K. Nazarpour "Pre-clinical application of abstract muscle synergies for myoelectric control," In: ISPO Trent International Prosthetic Symposium (TIPS). 2016, Glasgow: International Society for Prosthetics and Orthotics
Year(s) Of Engagement Activity 2016
URL http://www.ispo.org.uk/events_meetings-V.html
 
Description A talk at the BioNow conference (Newcastle) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Industry/Business
Results and Impact An event to bring together medical engineering research at Newcastle University and medical and biotechnology industry in North East England.
Year(s) Of Engagement Activity Pre-2006,2016
URL https://twitter.com/search?q=%23biofocus
 
Description A talk or presentation - Imperial College Surgical Society 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Undergraduate students
Results and Impact A talk about body integrated prosthetic devices, surgical challenges and skills required

For the link, please refer to the tweets on Jan 15
Year(s) Of Engagement Activity 2017
URL https://twitter.com/ICSMSurgicalSoc
 
Description A talk presented at Sheffiled University 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact The CATCH Network learned about machine learning in the control of prosthetic hands and Patient and Public Involvement. We were delighted to be joined by not one, not two, but three exciting speakers at the June 2016 CATCH Networking Lunch.
Year(s) Of Engagement Activity 2017
URL http://www.catch.org.uk/news-articles/6th-networking-lunch/
 
Description Future prosthetic: towards the bionic human 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact Kia Nazarpour gave an interview to The Engineer Magazine about the science-fiction vision of robotic prosthetic limbs that can be controlled by the brain and provide sensory feedback, which in his opinion is coming closer to reality.
Year(s) Of Engagement Activity 2018
URL https://www.theengineer.co.uk/future-prosthetic/
 
Description International Press Release 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Schools
Results and Impact Press Release on the Hand that Sees Project
Year(s) Of Engagement Activity 2017
URL http://www.bbc.co.uk/news/uk-england-tyne-39797011
 
Description International Press Release 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Media (as a channel to the public)
Results and Impact Press release about the UNESCO award received by the hand that sees project
Year(s) Of Engagement Activity 2018
URL http://www.ncl.ac.uk/press/articles/latest/2018/02/handthatseesunesco/
 
Description Interview 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact Gave an interview to The Engineer on the science-fiction vision of robotic prosthetic limbs that can be controlled by the brain and provide sensory feedback is coming closer.
Year(s) Of Engagement Activity 2018
URL https://www.theengineer.co.uk/future-prosthetic/
 
Description Interview with Dazed 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact Interview about "How AI could increase art world accessibility for disabled artists"
Year(s) Of Engagement Activity 2018
URL https://www.dazeddigital.com/art-photography/article/41334/1/how-ai-could-increase-art-world-accessi...
 
Description Interview with Verdict 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact This was an interview about how AI is exploding into healthcare
Year(s) Of Engagement Activity 2018
URL https://www.verdict.co.uk/ai-healthcare-growth-apps/
 
Description Newcastle High School for Girls Loves STEM event - A School Visit 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Schools
Results and Impact Showcasing bionics research in an STEM event in a local girls only school
Year(s) Of Engagement Activity 2017
 
Description School Visit 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Schools
Results and Impact Presenting control of myoelectric hands in a local girls only school
Year(s) Of Engagement Activity 2017
URL http://www.intellsensing.com/outreach-we-love-stem/
 
Description Speaking at IET Annual Lecture 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Postgraduate students
Results and Impact Presented challenges of myoelectric control in a public event
Year(s) Of Engagement Activity 2017
URL http://www.theiet.org/events/local/250075.cfm
 
Description The Future of Surgery 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Policymakers/politicians
Results and Impact Kia Nazarpour has given evidence to the Commission on the Future of Surgery, formed by Royal College of Surgeons. The purpose of the hearing was to set out a compelling and credible vision of the future advances in medicine and technology, including prosthetics, and how those developments will affect the delivery of surgical care in the United Kingdom. In particular, the Commission considers what the future of surgery is likely to look like for patients in five years, could look like in ten years, and might be in 15 to 20 years.
Year(s) Of Engagement Activity 2018
URL https://futureofsurgery.rcseng.ac.uk/report
 
Description Transhumanism 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact This was an interview by The Guardian about the future of Humans and how AI and bionics can create transhumans.
Year(s) Of Engagement Activity 2018
URL https://www.theguardian.com/technology/2018/may/06/no-death-and-an-enhanced-life-is-the-future-trans...
 
Description You have Been Upgraded 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact The team attened Manchester Science Festival to showcase our research in prosthetics control. Members of the public controlled a prosthetic hand with their muscle signals
Year(s) Of Engagement Activity 2018
URL https://www.manchestersciencefestival.com/event/you-have-been-upgraded/