A Cyber-Physical System for Unified Diagnosis and Treatment of Lung Diseases

Lead Research Organisation: University of Edinburgh
Department Name: Sch of Informatics


In this research, I will develop a cyber-physical system (CPS) for the diagnosis and treatment of lung diseases. My project is motivated by three facts:

1) Lung cancer treatment is most successful when it is found at an early stage. Treatment of early-stage cancer offers 73% chance of survival, whereas in late-stage this is reduced to 13%. However, the current methods for early diagnosis of peripheral lung lesions using bronchoscopic biopsy are challenging with varying diagnostic yield. Thus, there is a need to develop new technologies for reliable diagnosis of cancer in the lung periphery.

2) At present, surgical resection of malignant nodules (Tumours<3cm diameter) in the lung is the treatment modality of choice. However, most patients are not suitable surgical candidates, thus prompting the need for other therapeutic options. An emerging less invasive treatment option is bronchoscopic ablation of lung cancer via delivering physical therapy to lesions in the lung (cryotherapy/photonic ablation). Bronchoscopic ablation is not widely used to treat lung cancer, primarily due to the limited depth of penetration and small range of motion of the bronchoscope.

3) The overall in-hospital mortality rate for patients in ICU with ventilator associated events approaches 30%. New pulmonary infiltrates are a diagnostic challenge and due to the poor sampling methods available, patients are often initiated on non-targeted therapies. Bronchoscopy and sampling of the distal lung in diffuse diseases is not standardised and lacks repeatability and requires expert operators. The key aim of this research is to democratise ICU bronchoscopy and develop a platform using vision computing, EMT and external registration to enable non-skilled operators to sense, sample and diagnose pathology in vivo in situ.

To this end, I propose a CPS that addresses the unique issues of bronchoscopic diagnosis and treatment of lung diseases using novel mechatronic systems, control algorithms, and image guidance. The CPS has the potential to deliver unified diagnosis and treatment platform for early-stage lung cancer and on-site differential diagnosis of diffuse lung diseases by enhancing the current bronchoscopic technology in two specific ways:
1) Manipulation augmentation: I will design and develop a mechatronic device comprised of an active mini-bronch and a user interface for steering. The active mini-bronch is made of a flexible robot equipped with an endoscopic camera, fibre-optics for molecular imaging/sensing of tissue. It also provides a working channel that can be used for ablation of cancerous tissue or tissue sampling. The proposed mechatronic device can be used in two scenarios: (1) Autonomous tissue sampling in intensive care units: The mini-bronch uses control algorithms to navigate to lung subsegments and take multiple samples for diagnosis, (2) Semi-autonomous tissue diagnosis/treatment: the operator uses the user interface to navigate the mini-bronch to the periphery of the lung, characterise, sample and detect cancerous tissue, and then ablate . My control algorithms will provide unprecedented capabilities in terms of dexterity, safety, and ease of operation.
2) Visual augmentation: A key goal is to develop algorithms that employ emerging molecular imaging techniques to provide an imaging technology for in-vivo diagnosis of lung diseases.
I postulate that the proposed CPS will enable rapid on-site evaluation of lung diseases and successful bronchoscopic detection and therapy.

Planned Impact

In the past decade, there have been significant advances in technology for pulmonary medicine, including development of imaging technologies (e.g., endobronchial ultrasonography and Cone Beam CT), navigation platforms (e.g., electromagnetic navigation and virtual bronchoscopy), and ablation techniques (e.g., cryoablation, microwave ablation, and photonic ablation). However, the treatment of peripheral lung cancer still faces significant challenges that stem in part from the slow progression and limited capabilities of new bronchoscopic technologies. My proposed fellowship aims to address existing challenges by integrating technologies with a novel mechatronic mini-bronch and image-guided control algorithms.

The population most likely to benefit from the proposed research/innovations are those with suspected lung cancer and in the case of therapy, those who are not candidates for other common treatment options such as surgical removal of tumor or radiotherapy. Lung cancer is an insidious, smoking-related disease, and indeed many patients are not suitable candidates for surgery or radiotherapy, either due to the advanced stage of cancer or because of compromised cardiopulmonary function that precludes the risk of thoracic surgery. An emerging treatment option is a percutaneous ablation in which a needle carrying an ablation probe is inserted through the chest cavity to reach the lesions in the lung for ablation. Percutaneous ablation has a high complication rate, primarily pneumothorax (lung collapse), which makes such a technique difficult to pursue as an alternative treatment. The proposed bronchoscopic ablation methodology is a suitable alternative treatment option for patients with underlying lung disease, as it accesses cancer through the bronchial tree and there is a lower risk of pneumothorax, which is a significant concern in the medically inoperable.

The proposed technology will also make bronchoscopy accessible to new practitioners by offering augmented guidance and mechatronics assistance, allowing them to offer this minimally invasive treatment/diagnosis option to a greater number of patients. A particular target product profile is for the intensive care unit.

The results of this research have the potential for very significant impact on the field of flexible and continuum robotics. I expect the impact of the research/innovations to come directly or indirectly from the project outputs including:
1. Design of a mini-bronch by the integration of continuum robots
2. A mathematical framework for real-time modeling and visualisation of flexible continuum robots.
3. Open source algorithms for autonomous control and operator-in-the-loop control of flexible continuum robots.
These outputs will be of special interest to roboticists as they directly address the three grand challenges that have impeded the application of continuum robots in clinical practice: A) visualisation and integration; B) human-machine interaction; and C) shape sensing.

Another impact of this research will be on the people pipeline and skills base of the UK's healthcare engineering community. The proposed project has significant interdisciplinary aspects, connecting the robotics community and the EPSRC funded CDT in Robotics and Autonomous Systems at the University of Edinburgh to the medical research facilities at the Queen's Medical Research Institute (QMRI) supported by EPSRC IRC grant (PROTEUS project). 3 PDRAs and a Ph.D. student will be trained as independent researchers in the field of medical robotics, who will contribute to the knowledge base of medical robotics in the UK. Also, through the participation in workshops and Edinburgh's science festival we would aim to reach the general public, and in particular secondary school pupils, thereby encouraging them for careers in medical robotics.

Innovation, industrialisation and entrepreneurship are also at the heart of my fellowship proposal.
Description Severe lung injury that requires a ventilator is a common problem. During the Covid-19 pandemic, it was important to prioritize safe and effective sampling of the lungs to help develop better treatments and manage patients more effectively. Unfortunately, this technology wasn't available, so patients were often given broad-spectrum antibiotics that could lead to antimicrobial resistance. In this project, we developed a system that can deliver drugs directly to the lungs with incredible precision and then monitor the body's response. This technology could be used to treat future infectious diseases and expedite development of experimental drugs.
Exploitation Route The outcome of this research contributed to the opening of the Pandemic Science Hub to fight lung disease, A new multi-million pound research programme to develop treatments for lung infections such as Covid-19 and future pandemics has been announced at the University of Edinburgh with support from a significant donation by Baillie Gifford. The hub will combine Edinburgh's world-class ability to determine a person's genetic predisposition to lung injury (Roslin Institute) with advanced interventional robotics for drug delivery (this project), cutting-edge sensing and sampling technologies, and innovative clinical trial design (Queen's Medical Research Institute).

The Hub aims to quicken the discovery of new treatments. the team will deliver microdoses of multiple medicines to key areas of patients' lungs using the developed robotic system and observe if the drugs work on their own or in a combination. The constant risk of respiratory viruses, combined with the emergence of antibiotic resistance in respiratory diseases, means a radical new approach to streamlining drug development and evaluation is needed. To deliver this vision the hub will harness the expertise of the University's leading data scientists, roboticists, engineers, chemists, biologists, regulatory experts, drug developers, toxicologists, translational managers and clinicians.
Sectors Healthcare

URL https://www.ed.ac.uk/news/2022/pandemic-science-hub-to-fight-lung-disease
Description Enhancing Healthcare with Assistive Robotic Mobile Manipulation
Amount € 7,191,613 (EUR)
Funding ID 101017008 
Organisation European Commission 
Sector Public
Country European Union (EU)
Start 01/2021 
End 01/2024
Description IROS-SDC Travel Award
Amount ¥80,000 (JPY)
Organisation Institute of Electrical and Electronics Engineers (IEEE) 
Sector Learned Society
Country United States
Start 09/2022 
End 11/2022
Title Bronchoscopy Dataset For Visual Odometry 
Description We have developed a database of bronchoscopy videos in a phantom lung model and ex-vivo human lungs. The dataset contains 34 video sequences with over 23,000 frames. Odometry ground truth data (i.e., bronchoscope position in lung) is collected using a bespoke calibration rig and electromagnetic tracking sensors. Furthermore, we provide benchmark results for this dataset. State-of-the-art feature extraction algorithms including SIFT, ORB, Superpoint, Shi-Tomasi, and LoFTR are tested on this dataset 
Type Of Material Database/Collection of data 
Year Produced 2023 
Provided To Others? No  
Impact Bronchoscopy is a medical procedure that involves the insertion of a flexible tube with a camera into the airways to survey, diagnose and treat lung diseases. Due to the complex branching anatomical structure of the bronchial tree and the similarity of the inner surfaces of the segmental airways, navigation systems are now being routinely used to guide the operator during procedures to access the lung periphery. Current navigation systems rely on sensor-integrated broncho- scopes to track the position of the bronchoscope in real-time. This approach has limitations, including increased cost and limited use in non-specialized settings. To address this issue, researchers have proposed visual odometry algorithms to track the bronchoscope camera without the need for external sensors. However, due to the lack of publicly available datasets, limited progress is made. With our dataset, we will empower the robotics and machine learning community to advance the field. Furthermore, we provide benchmark results for this dataset using State-of-the-art algorithms and share our insights on challenges in endoscopic visual odometry. 
Description Collaboration between School of Informatics and Queen's Medical Research Institute 
Organisation University of Edinburgh
Department Queen's Medical Research Institute Edinburgh
Country United Kingdom 
Sector Academic/University 
PI Contribution Our research team brings expertise in medical device development and evaluation to the Queen's Medical Research Institute, which didn't exist before. We provide in-house design consultation on developing bespoke biomedical instruments, software development, and experiment design to remove technical barriers that frequently preclude the development and translation of new healthcare technology.
Collaborator Contribution As part of this collaboration, Queen's Medical Research Institute (QMRI) agreed to facilitate our cross-school collaboration through housing our equipment and providing laboratory space and office space in QMRI free of charge. Moreover, the collaboration gave us access to the Healthcare Technology Accelerator Facility (HTAF) (https://proteus.ac.uk/htaf/) at QMRI to help with regulatory pathways and ensure its compatibility with the current clinical workflow. HTAF is a non-profit initiative that provides a unique base for the development of healthcare technology (HT). HTAF has agreed to assist us throughout this award at all the stages of the research by providing professional consultancy from clinical experts and clinical project managers on in-human technology development, trial design, methodology, statistics, commercialization, and legal support.
Impact This is a multidisciplinary collaboration between engineers and computer scientists from the School of Informatics with Clinicians (Pulmonologists) and Photonics experts at the Queen's Medical Research Institute. This collaboration is relatively new and has not resulted in outcomes yet.
Start Year 2020
Title Library for Shape Estimation of Flexible Robots Using Single Point Position Measurement 
Description The software provides a fusion algorithm that combines a sensor measurement of the robot's pose with the mathematical model of a Flexible robot to estimate its overall shape. Python implementation of the shape estimation methodology presented in the following paper: "Shape Estimation of Concentric Tube Robots Using Single Point Position Measurement". E. Mackute, B. Thamo, K. Dhaliwal, M.Khadem. 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 
Type Of Technology Software 
Year Produced 2022 
Open Source License? Yes  
Impact Commonly, shape sensors are used to estimate an overall shape of a flexible robot in robotic-assisted minimally invasive surgeries such as bronchoscopy, colonoscopy, or percutaneous interventions. These sensors are expensive and heavily patented. As a result, most researchers in the medical robotics community rely on traditional sensors that provide measurements of position of a single point on the tip of the robot. This repo enables researchers to use single point measurements to estimate the overall shape of the flexible instrument, without relying on shape sensors. 
URL https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9982174
Title ROS Library for Simulation of Concentric Tube Robots 
Description Repo for modelling and simulation of Concentric Tube Robots in ROS. It implements the methodology presented in the following paper: "Data-driven Steering of Concentric Tube Robots in Unknown Environments via Dynamic Mode Decomposition ". Balint Thamo, David Hanley Kevin Dhaliwal, Mohsen Khadem. IEEE Robotics and Automation Letters, 2022. This repo uses C++ in ROS Melodic environment to simulate a CTR and includes a data-driven steering algorithm via the Dynamic Mode Decomposition method. You need to copy the folders into the src folder in your workspace. Then you can run it by the following command: roslaunch ctr_main ctr_main.launch 
Type Of Technology Software 
Year Produced 2022 
Open Source License? Yes  
Impact We developed the first publicly available simulation model for a flexible robot in Robot Operating System (ROS). Researchers can use this to test and validate their controllers. Previously, there was no such a platform and produced software for control of this robot could not be verified independently. 
URL https://doi.org/10.1109/LRA.2022.3231490
Description 2022 International Symposium on Medical Robotics 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact The 2022 Spring School on Medical robotics (SSMR) and the International Symposium on Medical Robotics (ISMR) was hosted on the campus of Georgia Tech, with workshops and tutorials held on jointly on April 19. ISMR featured a series of keynotes, semi-plenary talks, and workshops in addition to oral/poster presentations of original research.
The goal of ISMR is to bring together world-class researchers to present state-of-the-art research achievements and advances in surgical robotics, image-guided interventions, rehabilitation and assistive robotics, and prosthetics, as well as discuss challenges from the clinical point of view and address industry needs. It is envisioned that medical robots will integrate emerging technologies including soft robotics, smart materials, ergonomics, co-robotics, and artificial intelligence in the near future, and these topics will be featured in the symposium. Over 100 professionals and grad students attended.
Year(s) Of Engagement Activity 2022
URL https://ismr.gatech.edu/sites/default/files/program-PDFs/Booklet%202022_final.pdf
Description Annual Association of Coloproctology of Great Britain and Ireland (ACPGBI) meeting. 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Patients, carers and/or patient groups
Results and Impact I attended a debate on applications of colorectal robotic surgery. I was a panel member and engaged in discussions with clinicians, patients, and policymakers. The event was attended by more than 100 people. Several topics including current challenges and future of robotic surgery were discussed. We received feedback from end users (carers and patients). Throughout discussion, few key points were raised on the future of robotics and autonomy that changed practitioners views. Additionally, we received excellent feedback on ethical implications of robotics in clinics.
Year(s) Of Engagement Activity 2022
URL https://www.acpgbi.org.uk/events/2022_annual_meeting.aspx
Description Newton Fund Research Links on "Advanced Imaging Techniques for Pulmonary Disease" 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact The workshop aimed to: Support international development-relevant research; Contribute to capacity building of early career researchers; Establish new research links or significantly develop existing links, with the potential for longer-term sustainability.
It was attended by junior professionals and discussions were made on future collaborations on applications of robotics in Med tech, specifically image guided software.
Year(s) Of Engagement Activity 2021
URL https://blogs.ed.ac.uk/nfrlw/2021/
Description Pandemic Science Hub 
Form Of Engagement Activity A broadcast e.g. TV/radio/film/podcast (other than news/press)
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Media (as a channel to the public)
Results and Impact The Baillie Gifford Pandemic Science Hub was opened in 2022. It aims to use translational genomics - following clues from the human genome to identify, and rapidly test new treatments - with experimental medicine methods and robots to quickly evaluate and develop drugs for lung inflammation and injury caused by infection. It will combine Edinburgh's world-class ability to determine a person's genetic predisposition to lung injury with advanced interventional robotics for drug delivery, cutting-edge sensing and sampling technologies, and innovative clinical trial design.
Baillie Gifford is supporting the launch with a philanthropic gift of £14.7 million. The University aims to secure a total of £100m investment to accelerate discoveries to drive clinical translation in Covid-19 and other human lung diseases, as well as aiding preparedness for future pandemics.
The opening of the hub was broadcasted by Sky News.
Year(s) Of Engagement Activity 2022
URL https://news.sky.com/story/covid-19-scientists-receive-15m-boost-to-tackle-deadly-respiratory-illnes...
Description Scientific Workshop on Medical Robotics 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact I co-organised a workshop on "Holistic Integration of Design, Sensing, and Intelligence in Dexterous Surgical Robotic Systems". The workshop was held virtually on June 4, 2021.
Recently, we have seen increasing efforts aimed at implementing Dexterous Surgical Robotic Systems (DSRS) to push the frontiers of medical interventions. However, the deployment and integration of DSRS in real clinical scenarios have been hampered by issues such as limited sensory perception and safety concerns. In the workshop, we invited world-renown researchers and clinicians to discuss several possible avenues to address the challenges in the integration of DSRS, namely, (A) advanced sensing algorithms for robot perception, (B) control architectures and machine intelligence, and (C) novel designs and fabrication methods.
Year(s) Of Engagement Activity 2021
URL https://sites.google.com/view/icra2021workshop
Description Summer school of Surgical and Interventional Engineering (SIE) 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact I was an invited lecturer at 2022 Summer School on Surgical Robotics. the Summer School aimed to educate post-graduate research (PGR) students and research professionals on both technical and clinical advances in Surgical and Interventional Engineering. 40 internation pupils attended for a 5-day hybrid course that included a series of lectures on the topics of novel instrumentation for optimal surgery, and navigation algorithms for precise interventions. It further included hands-on projects with advanced surgical instrumentation setups.
Year(s) Of Engagement Activity 2022
URL https://www.kcl.ac.uk/short-courses/surgical-interventional-engineering
Description Workshop on "Safeguarding Public Health During Infectious Disease Pandemics Using Medical Robotics, Wearable Technology, and AI" 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact The goal of this workshop was to create awareness among engineers, scientists, and medical professionals concerning how the world can be better prepared for the next pandemic, reducing the negative impacts on healthcare systems, healthcare workers, and patients. Many interesting discussions were made on application of novel technologies in combating pandemics, which influenced the audience views including mine.
Year(s) Of Engagement Activity 2022
URL https://sites.google.com/ualberta.ca/2021-icee-workshop