Understanding Laparoscopic Skills for Robotic Training and Assistance
Lead Research Organisation:
Heriot-Watt University
Department Name: Sch of Engineering and Physical Science
Abstract
How do surgeons maintain the fine manipulations in a Minimally Invasive Surgery (MIS)? How do they get the feeling of how to move the hand, how much force to apply, and when? We know that surgeons gain these skills through years of training and experience. However, we do not have a complete understanding of "what is learned" and "what underlying mechanisms are developed" to guide manipulation of the hand or how to quantify these issues. This suggests some specific questions we need to address: what are the task-specific tactile/visual/sequential conditions that trigger specific hand-arm movements? How is hand-impedance controlled for these movements?
Training for conventional MIS with a co-manipulated robot (robotic training) and assistance for conventional MIS with a collaborative robot (robotic assistance) can significantly benefit surgeons as in both cases the natural haptic feedback to the surgeon is preserved. To this end, we need to understand how the hand and arm are used during conventional MIS. Hand-impedance and arm-muscle activity measurements are required to develop the quantitative knowledge of skilled manipulation. This knowledge can be used to build a framework that links arm muscle activation to hand-impedance control for fine force and position contact at the hand. Such a framework could offer insights in training level of surgeons and help support the lack of skilled personnel. In my global research agenda, which extends beyond this project, I aim to (i) identify skilled and unskilled type movements by monitoring the tool movement, (ii) measure the hand-impedance in complex movements with an interactive robot, (iii) measure arm muscle activity with electromyography (EMG) and find out muscle activation patterns of skilled manipulation, (iv) develop a robotic trainer with such knowledge, (v) develop a robotic assistant with impedance compensation using this knowledge. This project, which constitutes the backbone of my research agenda, covers the items (i), (ii), and partially (iv):
My goal in this project is to identify the MIS skills based on tool-position tracking, to perform hand-impedance measurements during the complex hand-manipulations, to develop a co-manipulated robotic trainer for MIS where the subject gets real-time feedback and assessment through tool movement tracking and hand-impedance measurements.
We will study three basic procedures in laparoscopic MIS: suturing, knot-tying, and bimanual carrying. The experimental setup will build upon my previous work - impedance measurements with an admittance controlled interactive robot - as well as my current work - on EMG based MIS skill identification and development of a mechanical adaptor to interface a MIS training platform to a robot. For the full research that directly relates to this project (items i, ii, iv), we need to understand on one hand, the skilled type tool manipulation and the corresponding hand-impedance levels (topic of this project), and on the other hand, the muscle activation patterns that generate these effects (topic of an accompanying PhD study). I will be contributing the accompanying PhD student to the proposed research, who is not funded by this project. The results of the work of this PhD student will be used in the funded project.
The robotic trainer will detect unskilled type movements and provide real-time feedback to notice those. The trainer will also assess the skill level of the subject by performing hand-impedance measurements besides evaluating tool movements. We will use a robot arm by attaching a MIS handle to its end-effector and an improved control/impedance measurement scheme that extends beyond my previous work. Subjects (professional-surgeons and novices) will perform MIS procedures with the tool attached to the end effector of the robot arm. The robot arm will passively follow and monitor the surgeon-hand movements and intervene from time to time for impedance measurements by introducing disturbances.
Training for conventional MIS with a co-manipulated robot (robotic training) and assistance for conventional MIS with a collaborative robot (robotic assistance) can significantly benefit surgeons as in both cases the natural haptic feedback to the surgeon is preserved. To this end, we need to understand how the hand and arm are used during conventional MIS. Hand-impedance and arm-muscle activity measurements are required to develop the quantitative knowledge of skilled manipulation. This knowledge can be used to build a framework that links arm muscle activation to hand-impedance control for fine force and position contact at the hand. Such a framework could offer insights in training level of surgeons and help support the lack of skilled personnel. In my global research agenda, which extends beyond this project, I aim to (i) identify skilled and unskilled type movements by monitoring the tool movement, (ii) measure the hand-impedance in complex movements with an interactive robot, (iii) measure arm muscle activity with electromyography (EMG) and find out muscle activation patterns of skilled manipulation, (iv) develop a robotic trainer with such knowledge, (v) develop a robotic assistant with impedance compensation using this knowledge. This project, which constitutes the backbone of my research agenda, covers the items (i), (ii), and partially (iv):
My goal in this project is to identify the MIS skills based on tool-position tracking, to perform hand-impedance measurements during the complex hand-manipulations, to develop a co-manipulated robotic trainer for MIS where the subject gets real-time feedback and assessment through tool movement tracking and hand-impedance measurements.
We will study three basic procedures in laparoscopic MIS: suturing, knot-tying, and bimanual carrying. The experimental setup will build upon my previous work - impedance measurements with an admittance controlled interactive robot - as well as my current work - on EMG based MIS skill identification and development of a mechanical adaptor to interface a MIS training platform to a robot. For the full research that directly relates to this project (items i, ii, iv), we need to understand on one hand, the skilled type tool manipulation and the corresponding hand-impedance levels (topic of this project), and on the other hand, the muscle activation patterns that generate these effects (topic of an accompanying PhD study). I will be contributing the accompanying PhD student to the proposed research, who is not funded by this project. The results of the work of this PhD student will be used in the funded project.
The robotic trainer will detect unskilled type movements and provide real-time feedback to notice those. The trainer will also assess the skill level of the subject by performing hand-impedance measurements besides evaluating tool movements. We will use a robot arm by attaching a MIS handle to its end-effector and an improved control/impedance measurement scheme that extends beyond my previous work. Subjects (professional-surgeons and novices) will perform MIS procedures with the tool attached to the end effector of the robot arm. The robot arm will passively follow and monitor the surgeon-hand movements and intervene from time to time for impedance measurements by introducing disturbances.
Planned Impact
Robotic assistance is studied by many groups, but robotic assistant for professional skill requiring tasks is a rather novel domain, which has the potential to shift the direction of robotic assistant studies. This is because, (i) the starting point of such study is understanding the tacit manipulation skills in order to answer what is actually needed to be assisted, and (ii) the study targets to develop novel robotic assistance methodologies, which are potentially difficult to devise without closely examining the human behaviours. This project and its results will have impacts on robotics research community, MIS training centres, industrial applications that require fine manipulation, and the general public view of robotics, through the following pathways.
1. Robotics Research Community - will learn about new methods of robotics assistance with human factors in the centre
We expect to have four top level journal publications out of this project, with the corresponding topics of "identification of skills through tool movement in MIS operations", "hand-impedance measurement for complicated hand movements", "robotic assistance with hand-impedance compensation in MIS", and "robotic training for hand movements in MIS", in top robotics journals. We will organize a workshop in one of the top robotics conferences with the topic of "skill training and assistance in minimally invasive surgery". We, as the Edinburgh Centre for Robotics, are organizing the EU Robotics Forum 2017 in Edinburgh and the PI is serving as the Finance Chair of the organizing committee. We will propose a session in one of the workshops in this Forum during the course of this project.
2. MIS training Centres - will be introduced to robotic training technology and will have more effective training systems
We envisage that this project will make a step change in the methods of MIS training. The robotic training methods will spread to other MIS skills training centres throughout UK. For that purpose, we will communicate our results and achievements to the other centres through the channels of medical community (e.g. Association of Laparoscopic Surgeons of Great Britain and Ireland, Royal College of Surgeons of Edinburgh, Royal College of Physicians and Surgeons of Glasgow, Royal College of Surgeons of England).
3. Industrial Applications - will benefit from robotic training-assistance technologies adaptable to various applications
This project will enable application of robotic assistance and training to a variety of actual industrial manipulation tasks. An examplary robotic training technology will be proposed to be developed in collaboration with welding training institutions in the UK. With dedicated visits to these institutions we will demonstrate the existent (and yet elementary) robotic training system for welding and the advanced training system applied to the MIS operations. The system will also be introduced to industrial audience through industry-oriented conferences (e.g. EU Robotics Forum) and publications.
4. General Public - will appreciate that collaborative-robots can enable people to do things better and that robots can be a companion for doing and learning things
The progress and results of this project will be proposed to be presented in public science events. It is remarkable that Edinburgh International Science Festival recently hosted two workshops devoted to "The Rights of The Machine" and "Unbottling the AI Demon", both of which revealed the public interest as well as cautiousness about robots taking part in daily life. The main cause of such cautiousness is that people think that the ever-developing robots are replacing human workers and creating an isolated automated sphere that humans do not have much understanding and impact. This project has the potential to challenge such fears as the robotic systems we develop are not to replace humans but to help them, and they do not function standalone but only in collaboration with humans.
1. Robotics Research Community - will learn about new methods of robotics assistance with human factors in the centre
We expect to have four top level journal publications out of this project, with the corresponding topics of "identification of skills through tool movement in MIS operations", "hand-impedance measurement for complicated hand movements", "robotic assistance with hand-impedance compensation in MIS", and "robotic training for hand movements in MIS", in top robotics journals. We will organize a workshop in one of the top robotics conferences with the topic of "skill training and assistance in minimally invasive surgery". We, as the Edinburgh Centre for Robotics, are organizing the EU Robotics Forum 2017 in Edinburgh and the PI is serving as the Finance Chair of the organizing committee. We will propose a session in one of the workshops in this Forum during the course of this project.
2. MIS training Centres - will be introduced to robotic training technology and will have more effective training systems
We envisage that this project will make a step change in the methods of MIS training. The robotic training methods will spread to other MIS skills training centres throughout UK. For that purpose, we will communicate our results and achievements to the other centres through the channels of medical community (e.g. Association of Laparoscopic Surgeons of Great Britain and Ireland, Royal College of Surgeons of Edinburgh, Royal College of Physicians and Surgeons of Glasgow, Royal College of Surgeons of England).
3. Industrial Applications - will benefit from robotic training-assistance technologies adaptable to various applications
This project will enable application of robotic assistance and training to a variety of actual industrial manipulation tasks. An examplary robotic training technology will be proposed to be developed in collaboration with welding training institutions in the UK. With dedicated visits to these institutions we will demonstrate the existent (and yet elementary) robotic training system for welding and the advanced training system applied to the MIS operations. The system will also be introduced to industrial audience through industry-oriented conferences (e.g. EU Robotics Forum) and publications.
4. General Public - will appreciate that collaborative-robots can enable people to do things better and that robots can be a companion for doing and learning things
The progress and results of this project will be proposed to be presented in public science events. It is remarkable that Edinburgh International Science Festival recently hosted two workshops devoted to "The Rights of The Machine" and "Unbottling the AI Demon", both of which revealed the public interest as well as cautiousness about robots taking part in daily life. The main cause of such cautiousness is that people think that the ever-developing robots are replacing human workers and creating an isolated automated sphere that humans do not have much understanding and impact. This project has the potential to challenge such fears as the robotic systems we develop are not to replace humans but to help them, and they do not function standalone but only in collaboration with humans.
People |
ORCID iD |
| Mustafa Suphi Erden (Principal Investigator) |
Publications
Gautier B
(2021)
Real-Time 3D Tracking of Laparoscopy Training Instruments for Assessment and Feedback.
in Frontiers in robotics and AI
Trute RJ
(2021)
Development of a Robotic Surgery Training System.
in Frontiers in robotics and AI
Tugal H
(2022)
Hand-impedance measurements with robots during laparoscopy training
in Robotics and Autonomous Systems
| Description | 1) We implemented an adaptive admittance control scheme that allows a transparent co-manipulation of robotic arms integrated into a laparoscopy training box in order to allow for hand-impedance measurements. This control scheme addresses the specific needs of an interactive robotic system integrated with the training setup, in the sense of adapting the admittance to maintain stable tool movement while occasionally in contact with the hard and soft environment within the training box. We performed experiments with professional laparoscopy surgeons and novice trainees and found more number of statistically significant difference in the left hand-impedance parameters compared to the right hand across the experts and the novices within the different level of training. An implication of this is the possibility that the left hand instrument (receiver) has been used more actively during needle placement and requires better control through both wrist and finger movements. Therefore, it seems that the movement of the receiver requires more skills than that of the driver (right-hand instrument). Statistically significant and consistent difference was observed particularly in the x direction, where the subjects were facing, while comparing the expertise levels of all the participant groups. Therefore, our findings support the idea that hand-impedance might be a measurable attribute of the skill level in laparoscopy, especially considering the orientation of the task with respect to the tools and the users, specifically the impedance in the direction (x) that the subject is facing. Among the mass, damping, and stiffness parameters that characterize a passive mechanical system, we found statistically significant differences with mass and damping. However, no statistically significant difference in the stiffness was observed. Stiffness is related to the active use of muscles and muscle strength. Our results indicate that the experts do not maintain skilled performance due to their muscle strength nor because they activate their muscles more than the others. Mass and damping parameters, on the other hand, are mostly structural parameters that depend on the orientation of the arm, rather than muscle activity. This means that the experts maintain a skilled activity mostly through better orienting their arm and upper body while using the instruments. This is an important finding that could inform laparoscopy training in order to optimally orient the arm and upper body during the operations. To the best of the authors knowledge, we have demonstrated, for the first time, hand-impedance measurements of the surgeons and novice subjects in the laparoscopic practice. The developed hand-impedance measurement technique and the identified values can be useful for the following purposes in future research: i) to inform laparoscopy training practices in order to optimally orient the arms to maintain optimal hand-impedance, ii) to be used in co-manipulated robotic trainers in order to gradually teach to the trainees the optimal hand-impedance, iii) to provide a biologically based method of assessment of laparoscopy skills with hand-impedance measurements, and iv) to be used in the stability analysis of the coupled robot-human system in a co-manipulated robotic trainer/assistant application. 2) We have developed an algorithm for tracking laparoscopy instruments in the video cues of a standard laparoscopy training box with a single webcam camera and developed new criteria to assess skill level using the extracted tool trajectories. The instrument tracking and assessment criteria we have developed together constitute a significant step towards developing a low cost, automated, and widely applicable laparoscopy training and assessment system using only video cues from a standard physical training box equipped with a webcam. The developed visual tracking algorithm recovers the 3D positions of the laparoscopic instruments tips to which simple colored tapes (markers) are attached. The new assessment criteria are based on frequency analysis and linear discriminant analysis of the 3D reconstructed trajectories of the instruments. The performance of these proposed criteria are compared to the conventional criteria for laparoscopy training and demonstrated to be superior on the data we have recorded from six professional laparoscopy surgeons and ten novice subjects. |
| Exploitation Route | Our study has develop fundamental knowledge about what skill relates to in laparoscopy. This knowledge can be used to develop novel training systems for laparoscopy. Our results inform the type of exercises that should be performed during laparoscopy training and novel assessment criteria. These might together speed up the training and render the assessments to be more objective and precise. |
| Sectors | Education,Healthcare |
| Description | The skill identification using impedance-measurements and muscle activity measurements were presented in various workshops and were appealing to a number of surgeons. Concrete connections have been established with laparoscopy surgeons/trainers and robotic surgeons to work on follow up project proposals. The robotic system for laparoscopic hand-impedance measurement that was developed within this project has been developed towards an experimental robotic surgery training system. This robotic surgery trainer was demonstrated to two different groups of surgeons (in Dundee and in Edinburgh) resulting in significant attraction and interest. As a result, currently (i) there is a PhD student, I am co-supervising with a surgeon from Dundee, who will be using this setup in their study; (ii) and a significant discussion of a joint project in collaboration with University of Edinburgh (both engineers and surgeons) and CMT Surgical Robotics company in Cambridge, with a visit planned to CMT, where the experimental robotic surgery training system will be in the center of the project. |
| Description | Cuschieri Skills Centre, Ninewells Hospital and Medical School, University of Dundee |
| Organisation | University of Dundee |
| Country | United Kingdom |
| Sector | Academic/University |
| PI Contribution | Heriot-Watt University has developed the robotic experimental setup to record hand-impedance, video streams, and EMG muscle activity of professional surgeons and trainee surgeons. The setup was brought to Cuschieri Skills Centre in Dundee and data were recorded with professional and trainee laparoscopy surgeons. |
| Collaborator Contribution | University of Dundee, specifically the Cuschieri Skills Centre associated with both Ninewells Hospital and Medical School and University of Dundee, made the arrangement of location, timing, and participants to perform laparoscopy training experiments and to record data with the robotic experimental setup. |
| Impact | Two conference paper resulted from this collaboration (one presented and published, one just accepted): 1) Hand-Impedance Measurement During Laparoscopic Training Coupled with Robotic Manipulators Tugal, H., Gautier, B., Kircicek, M. & Erden, M. S., 28 Jun 2018, (Accepted/In press) Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2018). IEEE 2) Laparoscopy instrument tracking for single view camera and skill assessment Gautier, B., Tugal, H., Tang, B., Nabi, G. & Erden, M. S., 31 Jan 2019, (Accepted/In press) 2019 IEEE International Conference on Robotics and Automation (ICRA). One journal paper submitted: 1) Hand-Impedance Measurement with Robots during Laparoscopy Training Harun Tugal, Benjamin Gautier, Benjie Tang, Ghulam Nabi, and Mustafa Suphi Erden 2-3 more Conference papers and 2 journal papers are in the pipeline to be prepared and submitted using the results of this collaboration. |
| Start Year | 2017 |
| Description | Edinburgh Science Festival - Robot-assisted surgery |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Public/other audiences |
| Results and Impact | This exhibition will take place in Edinburgh Science Festival in the venue the Summerhall on Monday 9 - Friday 13 April, 2018. The aim is to demo co-manipulated robotic system for laparoscopy training and to inform the public about co-manipulated robotic systems applied in medical domain for training and assistance purposes and to allow them interact with such a system in order to show how a human-and-robot can use a medical instrument together. The description is as follows: "Discover how robot assistants can help you perform keyhole surgery! You'll find demonstrations and explanations to help you find out about what it feels like to be a surgeon. You'll even have the chance to try out the laparoscopy training box, watching how robots can help make moving surgical instruments more accurate and precise. |
| Year(s) Of Engagement Activity | 2018 |
| URL | https://www.sciencefestival.co.uk/event-details/experimentarium-robot-assisted-surgery |
| Description | Industrial Partners of Edinburgh Centre for Robotics |
| Form Of Engagement Activity | Participation in an open day or visit at my research institution |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Industry/Business |
| Results and Impact | We demonstrated our robotized laparoscopy training system to the industrial partners of Edinburgh Centre for Robotics on the Industry Day (October 12th) of the Annual Conference of Edinburgh Centre for Robotics (October 11th-12th, 2017). The participants had the chance of having hands-on experience with our co-manipulated robots to use the laparoscopy instruments on a training box. With this demo, we show-cased that robots integrated with a training box through co-manipulation control can effectively used for measurement and assistance purposes while training novice subjects. |
| Year(s) Of Engagement Activity | 2017 |
| URL | https://www.edinburgh-robotics.org/news/201711/edinburgh-centre-robotics-annual-conference |