Exploring Human Hand Capabilities into Multifingered Robot Manipulation

Lead Research Organisation: University of Portsmouth
Department Name: Inst of Industrial Research

Abstract

It is evident that service robotics has the potential to improve people's quality of life and it holds the key to a number of unmet applications related to health care and rehabilitation. According to the prediction of International Federation of Robotics, the global market for intelligent service robots is forecast to reach 24.3 billion USD worldwide by 2010. A multi-fingered robotic hand is the most complex and dexterous robotic system, whose development represents frontiers in service robotics research. Recent innovations in motor technology and robotics have achieved impressive results in the hardware of robotic hands such as Robonaut hand. However, the manipulation systems of robotic hands are hardcoded to handle specific objects in specific ways, which significantly limits their transfer to a range of different situations and applications. The control and optimisation problems involved in robot hand manipulation are very difficult to solve in mathematical terms, however humans solve their hand manipulation related tasks easily using skill and experience. Object manipulation algorithms are required to meet the market requirement that robot hand systems should have human-like manipulation capabilities and be independent of robot hand hardware. Hence, the main challenge that researchers now face is how to enable robot hands to use what can be learned from human hands, to manipulate objects, with the same degree of skill and delicacy as human hands. The proposed work aims to investigate artificial intelligence (AI) methodologies and practical solutions which will allow robotic hands to automatically adapt to human environments and thus to enable them to autonomously perform useful manipulation tasks involved in daily living, pontentially for health care and rehabilitation applications. The investigation will focus on the following areas. 1) To generate a series of responsive human-like finger gaits for a robotic hand given an object to manipulate. This will have the capability to iteratively build a knowledge base representing the features of human hand manipulation behaviour and to efficiently provide corresponding robot hand gaits and manipulation strategies for a given manipulation task in a human environment.2) To develop feasible friction models for the interaction of objects and a robot/human hand. This will enable the application of existing mathematical research findings in multifingered robot manipulation to realworld applications in human environments and will integrate related methods in engineering and AI domains. 3) To develop an AI-based control architecture to ensure robust object manipulation of multifingered robots in terms of manipulation feasibility and efficiency. This will allow robot hands to perform stable human-like object grasping and manipulation and will also provide an open architecture which has the potential to introduce human brain (EEG/MRI signals) and human muscles (EMG signals) information into robotic hand systems.4) To validate the proposed algorithms by implementing these into a set of defined scenarios with a set of simulated multifingered robot hands and three different types of physical robot hands.
 
Description The investigation has achieved significant outcomes, which are provided according to the grant's objectives. 1) Fuzzy Gaussian Mixture Models have been provided to generate a series of responsive human-like finger gaits for a robotic hand given an object to manipulate. This enables a robotic hand the capability to iteratively build a knowledge base representing the features of human hand manipulation behaviour and to efficiently provide corresponding robot hand gaits and manipulation strategies for a given manipulation task in a human environment.2) Gaussian mixture models are employed to develop feasible friction models for the interaction of objects and a robot/human hand. This enables the application of existing mathematical research findings in multifingered robot manipulation to realworld applications in human environments and will integrate related methods in engineering and AI domains. 3) An AI-based control architecture is developed to ensure robust object manipulation of multifingered robots in terms of manipulation feasibility and efficiency. This allows robot hands to perform stable human-like object grasping and manipulation and will also provide an open architecture which has the potential to introduce human brain (EEG/MRI signals) and human muscles (EMG signals) information into robotic hand systems.4) the proposed algorithms are partially validated by implementing these into a set of defined scenarios with a set of simulated multifingered robot hands and three different types of physical robot hands.
Exploitation Route There are two ways: a) through commercial prosthetic hand companies, they transfer our technology to products or integrate to their existing products. There are two companies interested in this, they are Touch Bionics Ltd., UK, and Danyang Prosthesis Ltd. China; and there are two Universities who plan to employ our algorithms and approaches into their multi-fingered robotic hand systems, they are Shanghai Jiao Tong University, China, and Tokyo Metropolitan University, Japan. b) We currently discuss with Portsmouth University to set up a spin-out company to sell our technology/product. We have employed four ways to exploit the project outcomes: a) publish the project outcomes in peer-reviewed leading journals; b) organise international conferences or workshop to disseminate our project outcomes; c) organise leading journal special issues to overview the state of the art to attract more related researchers to our project outcomes; d) work closely with our collaborators so that they will maximise our project outcomes; e)work with potential industrial partners to seek for opportunity of transferring our technology to commercial products.
Sectors Digital/Communication/Information Technologies (including Software),Electronics,Healthcare

URL https://liuh.myweb.port.ac.uk/Projects.html
 
Description We have developed a multi-modal system integrating pressure, position, haptics sensors for hand motion analysis. This system outcome has demonstrated the research outcome is significant, it has been used to capture standard datasets. This system integrates pressure, position and haptics sensors, it provides a unified platform for analyzing hand motion. It's been used to capture standard datasets for hand-related algorithms development. Beneficiaries: Academic, industrial practitioners and algorithm developer. Contribution Method: This multi-modal system integrating pressure, position, haptics sensors for hand motion analysis, providing standard datasets for algorithms development in the world. We have developed a portable 16-Channell EMG system. This system has integrated our developed Gaussian mixture models, it provide many advantages including portability, high accuracy, and flexibility. This device has been further integrated with ultrasound sensors to detect the changes of the muscle shapes. This device can be used for measuring EMG-based hand manipulation gestures and tasks. Unlike existing devices, this equipment can capture EMG-based hand gestures at a higher accuracy. The ultrasound sensors provide complementary information for the hand motions, especially for the stroke patients whos muscle EMG-signals are weak. Beneficiaries: Academic, amputees, stroke patients, and the elderly people Contribution Method: To our best knowledge, this device is the first one which can measure EMG-based hand gestures and manipulation with such a high accuracy. The hardware design, combined with EMG and Ultrasound sensors, and integrated algorithms will have a significant impact on related research.
First Year Of Impact 2012
Sector Digital/Communication/Information Technologies (including Software),Electronics,Healthcare
Impact Types Societal,Economic

 
Description Collaboration with Prof. Naoyuki Kubota 
Organisation Tokyo Metropolitan University
Country Japan 
Sector Academic/University 
PI Contribution Provided a software package to control the robotic hand using learned human hand manipulation skills.
Collaborator Contribution Contributed a robotic hand to this research to evaluate and test the developed software package.
Impact The outcomes include papers published in conferences and journals, co-organised special sessions in international conferences and co-authored books.
Start Year 2010
 
Description Collaboration with Prof. Xiangyang Zhu 
Organisation Shanghai Jiao Tong University
Country China 
Sector Academic/University 
PI Contribution Provided a software package to control the prosthetic hand using learned human hand manipulation skills.
Collaborator Contribution Contributed a prosthetic hand (SJTU hand) to this research to evaluate the developed software package.
Impact The outcomes include papers published in conferences and journals, co-organised special sessions in international conferences and co-authored books.
Start Year 2010
 
Title A Multi-Modal System for Human Hand Motion Analysis 
Description In order to study and analyse human hand motions which contain multimodal information, a generalised framework integrating multiple sensors is proposed and consists of modules of sensor integration, signal preprocessing, correlation study of sensory information and motion intention identification. Three types of sensors are integrated to simultaneously capture the finger angle trajectories, the hand contact forces and the forearm electromyography (EMG) signals. 
Type Of Technology Systems, Materials & Instrumental Engineering 
Year Produced 2013 
Impact The proposed framework integrating the state- of-the-art sensor technology with the developed algorithms provides researchers a versatile and adaptable platform for human hand motion analysis and has potential applications especially in robotic hand or prosthetic hand control and Human Computer Interaction (HCI). 
 
Title Portable Multi-Channel EMG Capture System 
Description A custom-made sEMG sleeve system omitting the stage of muscle positioning is utilised to capture the sEMG signal on the forearm. A mathematic model for muscle activation extraction is established to describe the relationship between finger pinch forces and sEMG signal, where the genetic algorithm is employed to optimise the coefficients. 
Type Of Technology Systems, Materials & Instrumental Engineering 
Year Produced 2013 
Impact Based on this system, experimental results demonstrates that: 1) There is a systematical relationship between muscle activations and the pinch finger forces. 2) To estimate the finger force, muscle precise positioning for electrodes placement is not inevitable. 3) In a multi-channel EMG system, selecting specific combinations of several channels can improve the estimation accuracy for specific gestures. 
 
Description IEEE Transaction on Industrial Electronics Journal Issue on Intelligent Systems 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? Yes
Geographic Reach International
Primary Audience Other academic audiences (collaborators, peers etc.)
Results and Impact This special journal issue is to overview the state of the art in intelligent systems, it disseminates the EPSRC grant outcomes significantly.

During the journal issue preparation and review, details on call for papers, etc. have been well published in research communities and networks.

After the special issue, the latest research findings were identified and used to plan future research directions.
Year(s) Of Engagement Activity 2011
URL http://www.eng.auburn.edu/tie/ss09/Inteligent%20systems.pdf
 
Description IEEE Transactions on Industrial Informatics Speical Issue on Intelligent Video Systems 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? Yes
Geographic Reach International
Primary Audience Other academic audiences (collaborators, peers etc.)
Results and Impact This journal issue overviews the state of the art in intelligent video systems, it has significant impact in terms of disseminating the EPSRC grant outcomes.

During the special journal issue preparation and review process, lots of call for papers and communications have been delivered to our research communities.

The state-of-the-art and future challenges of the Intelligent Video Systems have been identified and highlighted, on which future related research plan will be based.
Year(s) Of Engagement Activity 2011
URL http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=6133476
 
Description Organising HSI2015 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact The grant holder organized 8TH INTERNATIONAL CONFERENCE ON HUMAN SYSTEM INTERACTION 2015 in Poland as one of general chairs. HSI is one of leading conference on human system interaction, which effectively improves this project impact and strengthens the outputs.
Year(s) Of Engagement Activity 2015
URL http://www.ieeesmc.org/conferences/calendar/event/232-8th-international-conference-on-intelligent-ro...
 
Description Organizing ICIRA2015 in Portsmouth 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact The grant holder organized 8TH INTERNATIONAL CONFERENCE ON INTELLIGENT ROBOTS AND APPLICATIONS 2015 in Portsmouth as a general chair. ICIRA is one of leading conference on intelligent robotic systems and applications, which effectively improves this project impact and strengthens the outputs.
Year(s) Of Engagement Activity 2015
URL http://www.ieeesmc.org/conferences/calendar/event/232-8th-international-conference-on-intelligent-ro...
 
Description Special Session in ROMAN 2015 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact The grant holder organised a Special Session on Cognitive Robotics at the 24th International Symposium on Robot and Human Interactive Communication, which is a significant contribution of the grant holder to disseminate/communicate this grant outcomes.
Year(s) Of Engagement Activity 2015
URL http://www.wikicfp.com/cfp/servlet/event.showcfp?eventid=43931
 
Description Special Session in SMC2015 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact The grant holder has been invited to give a conference special session in the annual conference of IEEE SMC society. This is a significant contribution of the grant holder to disseminate/communicate this grant outcomes.

IEEE-SMC is the leading conference in human machine systems, it's one of best venues to effectively deliver this project outcome and communication.
Year(s) Of Engagement Activity 2015
URL http://www.smc2015.org/sites/default/files/B20_cfp.pdf