Robotic Assisted Imaging

Lead Research Organisation: University College London
Department Name: Computer Science

Abstract

The paradigm of modern surgical treatment is to reduce the invasive trauma of procedures by using small keyhole ports to enter the body. Robotic assistant systems provide tele-manipulated instruments that facilitate minimally invasive surgery by improving the ergonomics, dexterity and precision of controlling manual keyhole surgery instruments. Robotic surgery is now common for minimally invasive prostate and renal cancer procedures. But imaging inside the body is currently restricted by the access port and only provides information at visible organ surfaces which is often insufficient for easy localisation within the anatomy and avoiding inadvertent damage to healthy tissues.

This project will develop robotic assisted imaging which will exploit the autonomy and actuation capabilities provided by robotic platforms, to optimise the images that can be acquired by current surgical imaging modalities. In the context of robotic assisted surgery, now an established surgical discipline, advanced imaging can help the surgeon to operate more safely and efficiently by allowing the identification of structures that need to be preserved while guiding the surgeon to anatomical targets that need to be removed. Providing better imaging and integration with the robotic system will result in multiple patient benefits by ensuring safe, accurate surgical actions that lead to improved outcomes.

To expose this functionality, new theory, computing, control algorithms and real-time implementations are needed to underpin the integration of imaging and robotic systems within dynamic environments. Information observed by the imaging sensor needs to feed back into the robotic control loop to guide automatic sensor positioning and movement that maintains the alignment of the sensor to moving organs and structures. This level of automation is largely unexplored in robotic assisted surgery at present because it involves multiple challenges in visual inference, reconstruction and tracking; calibration and re-calibration of sensors and various robot kinematic strategies; integration with surgical workflow and user studies. Combined with the use of pre-procedural planning, robotic assisted imaging can lead to pre-planned imaging choices that are motivated by different clinical needs.

As well as having direct applications in surgery, the robotic assisted imaging paradigm will be applicable to many other sectors transformed by robotics, for example manufacturing or inspection, especially when working within non-rigid environments. For this cross sector impact to be achieved the project will build the deep theoretical and robust software platforms that are ideally suited for foundational fellowship support.

Planned Impact

The impact of the proposed fellowship research will be widespread and has cross disciplinary potential. With the predicted uptake of robotic systems in the near future, this has applications in multiple sectors:

Healthcare: The focus for end-point outcomes of the fellowship project, is to improve real-time imaging during surgery by using the current systems for robotic prostate and kidney surgeries, as well as, the currently available intra-operative imaging modalities by developing computational support and capabilities. This has the potential to reduce surgical complications, patient readmission, disease specific indications such as kidney ischemia damage during vessel clamping, and generally all operative risks while increasing the success of surgery. These objectives, if achieved, will have a significant measureable impact on reducing mortality in patients and increasing the quality of life post-surgery, specifically for this fellowship in kidney and prostate cancers, but with potential applications in a wide range of clinical indications. Such impact would have obvious societal and economic measures.

Impact on Medical Technology: Facilitating new imaging during surgery has important synergies with surgical instrumentation, especially with emerging robotic systems. Multinational corporations in this space have recently made significant investments and acquisitions (Google, Johnson & Johnson - Ethicon, Medtronic, Stryker, and others) which are likely to result in a surge in surgical robot devices in the next five years. This will be a fertile ground for commercial exploitation of project outputs. The industrial partner, Intuitive Surgical, also provides a clear translational path within their platform, which is already used in over 500k procedures per year in different surgical specialisations. Advisory group steering will also help with links to medical device exploitation pathways.

Academic Impact: Computer vision techniques for real-time surgical procedures is a highly active and challenging area that is strongly represented at international conferences such as MICCAI, IPCAI and more recently ICRA and IROS by linking to robotics. Additional dissemination will happen at the leading vision meetings like CVPR, ECCV and ICCV for theoretical work on fundamental vision problems both in geometry (calibration and reconstruction) and inference (detection, tracking). The intersection of these areas incorporating sensor modelling, error propagation handling, highly novel nonlinear motion modelling, visual servo and control system integration will lead to publications in such high profile meetings as well as in high-impact journals (see Pathways to Impact and Cost of Support).

Impact on the Wider Research Community: Combination of imaging methodologies and robotics through i) data fusion, ii) multimodality systems, and iii) modelling and inverse problems, are recognised active topics, not only in surgical robotics and medical imaging, but more widely e.g. in seismology, non-destructive testing, nuclear maintenance and decommissioning, manufacturing and aerospace. Cross-fertilization between such different application areas is naturally stimulated through UCL Robotics, CMIC and the various other application focused vehicles throughout UCL. The Alan Turing Institute, in close proximity to UCL, will allow participation in wider programmes of research in large scale data processing exploiting theoretical developments in mathematics, signal processing and machine learning techniques. This and the EPSRC Networks will allow a rapid and effective mechanism to present project results.

Impact on teaching: By developing software throughout the project within structured frameworks, the project outputs will contribute to toolbox software that will be used within new and established programmes such as the MSc in Robotics and Computation and the MRes in Robotics.

Publications

10 25 50
publication icon
Allan M (2018) 3-D Pose Estimation of Articulated Instruments in Robotic Minimally Invasive Surgery. in IEEE transactions on medical imaging

publication icon
Bano S (2020) FetNet: a recurrent convolutional network for occlusion identification in fetoscopic videos. in International journal of computer assisted radiology and surgery

publication icon
Bano S (2020) Deep learning-based fetoscopic mosaicking for field-of-view expansion in International Journal of Computer Assisted Radiology and Surgery

publication icon
Biswas D (2017) Low-Complexity Framework for Movement Classification Using Body-Worn Sensors in IEEE Transactions on Very Large Scale Integration (VLSI) Systems

publication icon
Brandao P (2019) Widening siamese architectures for stereo matching. in Pattern recognition letters

 
Description Key research findings so far include new algorhtms to calibrate between camera and robotic coordinate frames dealing with different, new design of soft-robotic rails that can be used to support robotic assisted imaging in surgery, algorithms for ultra-fast reconstruction of the surgical site from video and also for understanding the site using high level segmentation, instrument tracking, video phase identification. These are all building towards a framework for better robotic assistance during surgery utilising imaging, robot control and interfacing to the surgeon.
Exploitation Route The work has led to a continuing collaboration with Intuitive Surgical Inc. (CA, USA) who support us with the da Vinci surgical system and access to its data as well as the dVRK platforms. Algorithms are typically made open source for dissemination through github and also we are continually making open datasets whenever possible for use by the community.
Sectors Healthcare

 
Description The research has led to multiple public speaking opportunities and public enagement events. We have published extensively to disseminate the work. Several patents have also been filed.
First Year Of Impact 2018
Sector Digital/Communication/Information Technologies (including Software),Healthcare
Impact Types Economic,Policy & public services

 
Description Actuated Robotic Imaging Skins
Amount £2,780,000 (GBP)
Organisation Royal Academy of Engineering 
Sector Charity/Non Profit
Country United Kingdom
Start 10/2019 
End 09/2029
 
Description CADDIE - Computer Aided Detection and Diagnosis for Intelligent Endoscopy
Amount £584,000 (GBP)
Organisation Innovate UK 
Sector Public
Country United Kingdom
Start 05/2019 
End 04/2021
 
Description EARTH SCAN
Amount £1,200,000 (GBP)
Organisation European Space Agency 
Sector Public
Country France
Start 11/2019 
End 10/2022
 
Description EndoMapper: Real-time mapping from endoscopic video
Amount € 3,697,227 (EUR)
Organisation European Commission 
Sector Public
Country European Union (EU)
Start 12/2019 
End 11/2023
 
Description IGT Network+
Amount £50,000 (GBP)
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 06/2017 
End 01/2018
 
Description MedCity Collaborate to Innovate
Amount £100,000 (GBP)
Organisation MedCity 
Sector Public
Country United Kingdom
Start 04/2017 
End 04/2018
 
Description NIHR i4i - Image-Guided Micro-Precise Flexible Robotic Tools for Retinal Therapeutics Delivery
Amount £1,017,363 (GBP)
Funding ID II-LB-0716-20002 
Organisation National Institute for Health Research 
Department NIHR i4i Invention for Innovation (i4i) Programme
Sector Public
Country United Kingdom
Start 08/2017 
End 08/2020
 
Description NIHR i4i Programme - A multi-modality, surgical planning and guidance system to improve the up-take of laparoscopic liver resection
Amount £1,300,000 (GBP)
Funding ID II-LA-1116-20005 
Organisation National Institute for Health Research 
Department NIHR i4i Invention for Innovation (i4i) Programme
Sector Public
Country United Kingdom
Start 01/2018 
End 01/2021
 
Description Simulation of Complex Off-Road Environments for Autonomous Vehicle Development
Amount £929,000 (GBP)
Organisation Innovate UK 
Sector Public
Country United Kingdom
Start 04/2019 
End 03/2021
 
Description UKRI AI Centre for Doctoral Training in Foundational Artificial Intelligence
Amount £6,443,206 (GBP)
Funding ID EP/S021566/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 03/2019 
End 09/2027
 
Description p-Tentacle: Pneumatically Attachable Flexible Rails For Surgical Applications
Amount £99,000 (GBP)
Funding ID EP/R511638/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 04/2019 
End 03/2020
 
Title Depth from Endoscopy 
Description The database containts a set of paired images from a colonoscopy/endoscopy simulator that generates RGB images and also depth images. The data can be used to train models to infer depth from RGB and extended to infer the motion of the camera and the map of the endoluminal environment. 
Type Of Material Database/Collection of data 
Year Produced 2019 
Provided To Others? Yes  
Impact We are seeing groups world wide beginnging to use the dataset both in academie and in industry (recent Google Health publication). 
URL http://cmic.cs.ucl.ac.uk/ColonoscopyDepth/
 
Title Surgical instrument database of video 
Description Database of videos with surgical instruments within a lab setting. Useful for computer vision and image analysis development. 
Type Of Material Database/Collection of data 
Year Produced 2016 
Provided To Others? Yes  
Impact There is now an active challenge at MICCAI running on this topic. It is organised by the student, Max Allen, who developed the initial concepts. 
URL http://www.surgicalvision.cs.ucl.ac.uk/resources/benchmarking/#home
 
Description Collaboration with Odin Vision Ltd 
Organisation Odin Vision
Sector Private 
PI Contribution Research and supervision of PhD students to develop novel AI technologies for endoscopy.
Collaborator Contribution Provision of infrastructure for video data processing, storage and labelling to develop AI models. Funding for PhD studentships.
Impact No outputs yet as the collaboration has just begun.
Start Year 2020
 
Description Intuitive Surgical CA 
Organisation Intuitive Surgical Inc
Country United States 
Sector Private 
PI Contribution Development of algorithms and AI models to process video and other robotic data from the Intuitive Surgical system in clinical environments.
Collaborator Contribution Access to data from the Intuitive Surgical system.
Impact N/A
Start Year 2013
 
Description da Vinci Research Kit Consortium partnership 
Organisation Intuitive Surgical Inc
Country United States 
Sector Private 
PI Contribution We have contributed in the development of computer vision software and algorithms for stereoscopic endoscope video.
Collaborator Contribution Donation of equipment and exchange of knowledge and access to data.
Impact The partnership and consortium have resulted in funding from NIH, multiple papers, knowledge exchange and student engagement as well as dissemination activities.
Start Year 2016
 
Title Method And Apparatus For Estimating The Value Of A Physical Parameter In A Biological Tissue 
Description A method and apparatus are provided for estimating the value of a physical parameter of biological tissue. The method comprises acquiring a colour image of the biological tissue from a single image capture device; extracting from the colour image at least two images in respective optical wavebands having a different spectral sensitivity from one another, whereby a given location in the biological tissue is present in each of the extracted images; providing a physical model of the optical properties of the biological tissue, wherein the optical properties of the biological tissue are sensitive to the value of said physical parameter; and estimating the value of the physical parameter at said given location based on an intensity value at that location for each extracted image. The estimating utilises the physical model of the optical properties of the biological tissue and the spectral sensitivity for each respective waveband. 
IP Reference US2019320875 
Protection Patent granted
Year Protection Granted 2019
Licensed No
Impact No impacts at present.
 
Title CADDIE - Computer Aided Detection and Diagnosis for Intelligent Endoscopy 
Description CADDIE - Computer Aided Detection and Diagnosis for Intelligent Endoscopy - is an Innovate UK funded project to put Odin Vision Ltd (spin out from UCL) technology into a clinical trial at UCLH. The company's first product utilises AI to assist the detection and characterisation of polyps during video colonoscopy. 
Type Support Tool - For Medical Intervention
Current Stage Of Development Early clinical assessment
Year Development Stage Completed 2019
Development Status Under active development/distribution
Impact Similar technology is being developed to support interventions in upper GI endoscopy. 
 
Title Surgical robot arm visualization software 
Description Cinder application designed to enable visualisation of robotic arms at a given configuration overlaid on a camera feed. This is particularly useful for surgical robotics where we can estimate arm position from joint kinematic data and then render the joints on the camera's field of view. 
Type Of Technology Software 
Year Produced 2016 
Open Source License? Yes  
Impact Multiple research sites have used it for development and testing of surgical vision algorithms involving robotic instruments. 
URL http://www.surgicalvision.cs.ucl.ac.uk/resources/code/
 
Company Name ODIN MEDICAL LIMITED 
Description Odin specialises in developing computer aided diagnostic solutions for endoscopic diagnostic and therapeutic procedures. 
Year Established 2018 
Impact Too early to report key achievements.
 
Description Endoscopic Imaging and AI - Artificial Intelligence in Medicine, Enterprise Network Europe, London, UK 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Industry/Business
Results and Impact Dr. Stoyanov gave a talk at an event organised to facilitate exchange and new business opportunities between the UK and Austria co-organised with the Austrian Trade Commission. The event was well attended and involved interesting discussion on how AI will influence clinical decision making.
Year(s) Of Engagement Activity 2018
 
Description Interview for Korean national newspaper 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Public/other audiences
Results and Impact Dr. Danail Stoyanov was interviewed by one of the largest Korean newspapers on the role of robotics in surgery and healthcare.
Year(s) Of Engagement Activity 2017
URL http://mnews.joins.com/article/2200741
 
Description Keynote talk at CARE Workshop, MICCAI 2018, Quebec, Canada 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Dr. Stoyanov gave a keynote talk at the Computer Assisted and Robotic Endoscopy workshop held at MICCAI 2018. The talk had a lively discussion afterwards focusing on the future impact of endoscopic computer vision in surgery.
Year(s) Of Engagement Activity 2017
 
Description London Open House @ Here East 
Form Of Engagement Activity Participation in an open day or visit at my research institution
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact London Open House @ Here East
Year(s) Of Engagement Activity 2018
 
Description Talk at the Winter Meeting of the European Association of Endoscopic Surgeons (EAES) 2018 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Dr. Danail Stoyanov gave a talk on the application of AI in endoscopic imaging, which invoked an interesting discussion on translation and potential clinical opportunities.
Year(s) Of Engagement Activity 2018
 
Description UCL Being Human Event 'Exploring Under the Skin' 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Other audiences
Results and Impact Talks and an exhibition and interactive exploration of how the humanities, arts and sciences have separately and collectively come together in the endeavour of exploring under our skin and our own reactions to this, from artworks, to the curiosity cabinets of the 1700s and through to modern ways that medicine and the humanities collide.
Year(s) Of Engagement Activity 2018
URL https://beinghumanfestival.org/event/exploring-under-the-skin/