Turing AI Fellowship: Reinforcement Learning for Healthcare
Lead Research Organisation:
Imperial College London
Department Name: Bioengineering
Abstract
In this Turing Artificial Intelligence Acceleration Fellowship, I will focus on artificial intelligence for medical treatments and therapies. I take the view that AI is a question on how to realise artificial systems that solve practical problems currently requiring human intelligence to solve, such as those solved by clinicians, nurses and therapists. Critical care is high risk and highly invasive environment caring for the sickest patients at greatest risk of death. Patients within this environment are highly monitored, enabling sudden changes in physiology to be attended to immediately. In addition, this monitoring requires a heavier staffing ratio (often 1:1 nursing; 1:8 medical) and variances in human factors and non-technical pressures (e.g. staffing, skill-mix, finances) leads to critical care delivery being disparate.
AI in healthcare is a hard problem as, due to the diversity and variability of human nature, systems have to cope with unexpected circumstances when solving perceptual, reasoning or planning problems. Crucially, AI has two facets: Understanding from data, and Agency. While rapid strides have been made on learning from data, e.g. how to make medical diagnosis more precise and faster than human experts, there is little work on how to carry on after the diagnosis, e.g. which therapy and treatment to conduct. The latter requires agency and has seen fewer applications as it is a harder problem to solve.
My clinical partners and I want to develop the required AI algorithms that can learn and distil the best plan of action to treat a specific patient, from the expert knowledge of clinicians. We will focus on an area of AI called RL that has been successful in enabling robots and self-driving cars to learn a form of autonomous agency. We want to transform these methods into the healthcare domain. This will require the development of new RL algorithms, able to efficiently understand the state of a patient from noisy and ambiguous hospital data. The system will not only learn to recommend interventions such as prescribing drugs and changing dosages as needed per patient but to make these recommendations in a manner that is meaningful to the clinical decision-makers and helps them make the best final decision on a course of action.
The methods developed as part of this project can be used in different applications beyond healthcare. Many sectors within industry, such as aerospace, or energy, deal with similar bottlenecks. These are highly regulated environments, with great need for decisions making support, but a scarcity of highly skilled human experts. With sufficient data, our methods can be applied to these sectors as well, to distil the required human expertise and best practices from top experts, and use them to drive decision making all over the sector, for increased efficiency and safety.
AI in healthcare is a hard problem as, due to the diversity and variability of human nature, systems have to cope with unexpected circumstances when solving perceptual, reasoning or planning problems. Crucially, AI has two facets: Understanding from data, and Agency. While rapid strides have been made on learning from data, e.g. how to make medical diagnosis more precise and faster than human experts, there is little work on how to carry on after the diagnosis, e.g. which therapy and treatment to conduct. The latter requires agency and has seen fewer applications as it is a harder problem to solve.
My clinical partners and I want to develop the required AI algorithms that can learn and distil the best plan of action to treat a specific patient, from the expert knowledge of clinicians. We will focus on an area of AI called RL that has been successful in enabling robots and self-driving cars to learn a form of autonomous agency. We want to transform these methods into the healthcare domain. This will require the development of new RL algorithms, able to efficiently understand the state of a patient from noisy and ambiguous hospital data. The system will not only learn to recommend interventions such as prescribing drugs and changing dosages as needed per patient but to make these recommendations in a manner that is meaningful to the clinical decision-makers and helps them make the best final decision on a course of action.
The methods developed as part of this project can be used in different applications beyond healthcare. Many sectors within industry, such as aerospace, or energy, deal with similar bottlenecks. These are highly regulated environments, with great need for decisions making support, but a scarcity of highly skilled human experts. With sufficient data, our methods can be applied to these sectors as well, to distil the required human expertise and best practices from top experts, and use them to drive decision making all over the sector, for increased efficiency and safety.
Organisations
- Imperial College London (Lead Research Organisation)
- Siemens AG (Collaboration)
- Shell Global Solutions International BV (Collaboration)
- Airbus Group (Collaboration)
- Siemens AG (International) (Project Partner)
- Imperial College Healthcare NHS Trust (Project Partner)
- UNIVERSITY COLLEGE LONDON (Project Partner)
- Shell Research UK (Project Partner)
- Airbus Defence and Space GmbH (Project Partner)
Publications
Alam MS
(2022)
Repurposing of existing antibiotics for the treatment of diabetes mellitus.
in In silico pharmacology
Currie SP
(2022)
Movement-specific signaling is differentially distributed across motor cortex layer 5 projection neuron classes.
in Cell reports
Faisal AA
(2021)
Putting touch into action.
in Science (New York, N.Y.)
Festor P
(2022)
Assuring the safety of AI-based clinical decision support systems: a case study of the AI Clinician for sepsis treatment.
in BMJ health & care informatics
| Description | The aim of my Turing AI Fellowship is to develop reinforcement learning-based AI for human interventions. Core of this work is the idea to use off-line reinforcement learning (the AI Clinician) to model the state of a person (here an intensive care patient) and learn from historical data how human expert interventions (here an intensive care doctor) how to treat patients better.The core of the grant is focussed on healthcare (the so called AI clinician) and industry partners (Shell, Airbus) bring in applications to other domains of industry and services. The active clinical trials on the AI Clinician (which is now an NIHR/NHS flagship project and deployed in 4 London hospitals) drives a lot of the underpinning AI work that is needed beyond As the first example of clinically trialled AI-learned Digital Therapeutics our work has by now been replicated in over 30 independent studies. The first, foundational problem we cracked (Li & Faisal, 2021) required methods to handle the fact that hospital data are only painting a partial observable picture of the patient and we required the ability to handle continuous state and action spaces in near real-time, while representing uncertainty that the model had - which resulted in the called Bayesian Distributional Policy Gradients (AAAI, 2021) which combines a Bayesian RL model for parameter uncertainty and a distributional RL model for return uncertainty in a principled manner with a combined posterior. A clear lesson learned from the clinical evaluation of the RL agent was that we needed better methods for a. uncertainty quantification from offline RL and developed methods to estimate epistemic & aleatoric components (to communicate risk to clinicans) (Festor et Faisal, 2021, ICML MLHC) , b. Handling hierarchical decomposition of the policy (for explainabiltiy purposes), c. Safety-critical aspect (to ensure safety of the learned decisions) (Festor et al, 2022 BMJ HIC) and d. Methods that allow us to empirically measure which and what explanations and AI recommendations had the biggest impact on clinical decision making (Shafti et Faisal, 2022), e. The ethics of AI decisions on humans (Post, Brett & Faisal, 2022, AI & Ethics). f. Developing methods that combine federation & transfer learning to learn from data across different sites. Crucially, these developments are now been combined into forthcoming major paper that define AI Clincian 2.0 which is going into clinical testing this year. Follow on funding allows us to take the AI Clinician to paediatric intensive care. We also expaneded the application base for this digital therapeutics work (pandemic-wide intensive care work was slow), and so we expanded this into settings where we found more capacity to apply ourselves namely the treatment of neurological disorders. This required the development of appropriate patient state modelling that was on par with the intensive care work (Kardivelu et Faisal, Nature Medicine 2023; Ricotti et Faisal, Nature Medicine 2023) before we could apply these to reinforcement learning based neurological interventions (eg Wannawas & Faisal, 2021,2022,2023). In addition this personal fellowship was meant to accelerate Prof Faisal's career, this has been realised by Prof Faisal having been named as a new Director of the Alan Turing Institute, the national AI research lab. |
| Exploitation Route | The AI Clinician is now in active clinical trials with end-users (intensive care clinicians) and in 4 hospitals across London (St Marys, Charing Cross, Hammersmith & UCLH). There is a glboal community of over 30 research groups using our models and data processing to develop the AI Clinician idea further. |
| Sectors | Communities and Social Services/Policy Digital/Communication/Information Technologies (including Software) Energy Healthcare Manufacturing including Industrial Biotechology Pharmaceuticals and Medical Biotechnology Transport Other |
| URL | https://aiclinician.icu |
| Description | The active clinical trials (the AI Clinician is now an NIHR/NHS flagship project and being deployed in 4 London hospitals since 2022 - the fellowship drives a lot of the underpinning AI work that is needed beyond As the first example of clinically trialled AI-learned Digital Therapeutics our work has by now been replicated in over 30 independent studies. Clinical sites from the US to the Netherlands are now deploying the AI clinician code/software stack/data selectors for their own trials. |
| First Year Of Impact | 2021 |
| Sector | Aerospace, Defence and Marine,Education,Energy,Healthcare |
| Impact Types | Cultural Economic Policy & public services |
| Description | AI for decoding and executing human intentions |
| Geographic Reach | Multiple continents/international |
| Policy Influence Type | Participation in a guidance/advisory committee |
| Impact | Changes in regulatory pipelines have been evaluated and discussed in response to our inputs and are in active change. |
| Description | BBSRC AI strategy |
| Geographic Reach | National |
| Policy Influence Type | Contribution to a national consultation/review |
| Impact | Advised BBSRC on AI strategy which led to new BBSRC AI call. |
| Description | MRC AI strategy panel |
| Geographic Reach | National |
| Policy Influence Type | Contribution to a national consultation/review |
| Impact | Guidance recommendation for MRC Biostatistics Unit strategy |
| Description | Airbus Space & Defense - Airbus grant to support Turing AI Fellowship |
| Amount | £100,000 (GBP) |
| Organisation | Airbus Group |
| Sector | Academic/University |
| Country | France |
| Start | 01/2021 |
| End | 12/2023 |
| Description | Rosetrees Interdisciplinary Award |
| Amount | £300,000 (GBP) |
| Funding ID | Only one prize award is awarded every year. |
| Organisation | Rosetrees Trust |
| Sector | Charity/Non Profit |
| Country | United Kingdom |
| Start | 09/2023 |
| End | 09/2027 |
| Description | Shell - PostDoc research grant to support Turing AI Fellowship |
| Amount | £408,727 (GBP) |
| Organisation | Shell Centre |
| Sector | Private |
| Country | United Kingdom |
| Start | 01/2021 |
| End | 12/2025 |
| Description | Skyrmionics for Neuromorphic Technologies |
| Amount | £703,710 (GBP) |
| Funding ID | EP/V028189/1 |
| Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
| Sector | Public |
| Country | United Kingdom |
| Start | 09/2021 |
| End | 09/2024 |
| Description | Wearable Deep Spinal Interfacing (WDSI) |
| Amount | £202,026 (GBP) |
| Funding ID | EP/X018466/1 |
| Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
| Sector | Public |
| Country | United Kingdom |
| Start | 11/2022 |
| End | 04/2024 |
| Description | Airbus Partnership in Machine Learning |
| Organisation | Airbus Group |
| Department | Airbus Defence and Space UK |
| Country | United Kingdom |
| Sector | Private |
| PI Contribution | Airbus Space & Defense want to collaborate with on translation of my foundational AI on reinforcement learning for human-in-the-loop control to work in the aerospace domain, specifically Robotics applications. The support included a jointly supervised PhD student and the time and efforts of the Airbus Data R&D Science and Robotics team in the UK. The collaboration was supported now by a new Airbus CASE PhD studentdship. |
| Collaborator Contribution | Funding for a PostDoc to kickstart a project. |
| Impact | Publications will be listed in the publication section. |
| Start Year | 2021 |
| Description | Partnership with Shell on Reinforcement Learning |
| Organisation | Shell Global Solutions International BV |
| Department | Shell Research Ltd |
| Country | United Kingdom |
| Sector | Private |
| PI Contribution | The purpose of this is to engage a Postdoctoral researcher and translate the impact of the Turing AI Fellowship's foundational AI work into the energy sector application domain. This is to work directly with Daniel Jeavons, the General Manager of Data Science at Shell Research in addition, Shell commit to working with me directly through discussion meetings on a monthly basis as an in-kind contribution to support the Fellowship. Since the start of this partnership the collaboration was enriched by a further Shell CASE studentship. |
| Collaborator Contribution | Funding for PostDocs and an EPSRC CASE studentship. |
| Impact | Publications will be listed under the publication section. |
| Start Year | 2021 |
| Description | Siemens Parternship in Digital Twins |
| Organisation | Siemens AG |
| Country | Germany |
| Sector | Private |
| PI Contribution | Dr Annemarie Frie, a Head of Siemens corporate research (aka the AI and robotics research division) commit to working with me directly through monthly discussion meetings as an in-kind contribution to support the Fellowship. For |
| Collaborator Contribution | see above. |
| Impact | Publications will be listed in the publication seciton |
| Start Year | 2021 |
| Title | AI Clinician XP2 |
| Description | The cornerstone of sepsis resuscitation is the administration of intravenous fluids (IVF) and/or vasopressors (drugs that squeeze the blood vessels to increase blood pressure) to maintain blood flow to prevent organ failure. However, there is huge uncertainty around the individual dosing of these drugs in an individual patient, partially due to high sepsis heterogeneity. The current guidelines provide recommendations at a population-level but fail to individualise the decisions. Wrong decisions lead to poorer outcomes and increased ICU-resource use. A tool to personalise these medications could improve patient survival. We have developed a new method to automatically and continuously review and recommend the correct dose of these medications to doctors, which was created using artificial intelligence (AI) techniques applied to large medical databases. The method we used is called reinforcement learning, and we call the technology the "AI Clinician". |
| Type | Support Tool - For Medical Intervention |
| Current Stage Of Development | Early clinical assessment |
| Year Development Stage Completed | 2024 |
| Development Status | Under active development/distribution |
| Clinical Trial? | Yes |
| Impact | This present experiment will test the feasibility of running the AI Clinician in real-time in operational ICUs, in preparation for a future large scale multicentre randomised trial that will test for an improvement in clinically relevant outcomes. At this stage and in the interest of focusing on prescribers first, we will only be testing the use of the system by ICU doctors. Studies with nurses will be conducted in the future. |
| Title | Passive Evaluation in Operational Environment of the AI Clinician Decision Support System for Sepsis Treatment (AI Clinician XP1) |
| Description | We have developed a new method to automatically and continuously review and recommend the correct dose of these medications to doctors, which was created using artificial intelligence (AI) techniques applied to large medical databases. The method we used is called reinforcement learning, and we call the technology the "AI Clinician". In thisAI Clinician XP1 trial, we tested the safety of the AI Clinician when running in "shadow mode", i.e. in pseudonymised batches of patient data presented to off-duty ICU clinicians. This enabled us to 1) develop methods and software to connect to real-time electronic health records (EHR); 2) check the safety of the algorithm when used in a contemporary UK ICU patient cohort. |
| Type | Support Tool - For Medical Intervention |
| Current Stage Of Development | Refinement. Clinical |
| Year Development Stage Completed | 2024 |
| Development Status | Under active development/distribution |
| Clinical Trial? | Yes |
| Impact | We want to test the feasibility of running the AI Clinician in real-time in the ICU of 4 sites across a period of 6 months and identify patients suitable for the system. |
| Company Name | Ethomix Ltd |
| Description | |
| Year Established | 2024 |
| Impact | The company was recently formed, in the run up to the formation the company won several incubator and startup awards, and is now getting ready to trade with their first customer. |
| Description | BBC exclusive feature on our work (including TV, news, documentary and web features) |
| Form Of Engagement Activity | A press release, press conference or response to a media enquiry/interview |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Public/other audiences |
| Results and Impact | Exclusive coverage of our work in BBC |
| Year(s) Of Engagement Activity | 2023 |
| URL | https://www.bbc.co.uk/news/science-environment-64326125 |
| Description | Co-organiser of the International Conference on Complex Acute Illness (ICCAI) 2024 in Bethesda (MD, USA) |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | I have become a co-organiser and been nominated as board member of the leading serious and complex illness society and their annual conference (ICCAI). |
| Year(s) Of Engagement Activity | 2023,2024 |
| URL | https://iccai.org |
| Description | Disability in AI (talk) |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Public/other audiences |
| Results and Impact | A public engagement talk on Disability, AI and Assitive Technology at UCL |
| Year(s) Of Engagement Activity | 2022 |
| Description | IEMTRONICS 2023 Keynote |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Keynote at technology conference |
| Year(s) Of Engagement Activity | 2023,2024 |
| Description | Industry Keynote at FZI Karsruhe |
| Form Of Engagement Activity | Participation in an open day or visit at my research institution |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Keynote at Forschungszentrum Informatik (FZI) in Karlsruhe (Germany) one of Germany's major private-public industry research institutes. |
| Year(s) Of Engagement Activity | 2023,2024 |
| Description | Keynote at Annual All-German-Medical-School conference (NUM) |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Keynote at key event held in Berlin at which all German medical schools participated. |
| Year(s) Of Engagement Activity | 2024 |
| URL | https://www.linkedin.com/posts/drgoeschl_num-convention-2024-activity-7154096014960771072-w_PO/ |
| Description | Keynote at Open Medical Forum held at TU Munich |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Keynote at major medical event held in Munich. |
| Year(s) Of Engagement Activity | 2023,2024 |
| URL | https://www.tum.de/en/news-and-events/all-news/press-releases/details/the-importance-of-a-transdisci... |
| Description | Keynote at annual governing political party closed-door retreat |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Policymakers/politicians |
| Results and Impact | Gave keynote at an annual governing party retreat that has lead to government decisions and cabinet decisions. |
| Year(s) Of Engagement Activity | 2023,2024 |
| Description | Keynote at the American Association for the Advancement of Science (AAAS) in Washington DC 8/11/2023 |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Public/other audiences |
| Results and Impact | Keynote at global open forum on research in Washington, DC (USA). |
| Year(s) Of Engagement Activity | 2023,2024 |
| URL | https://www.science.org/content/webinar/ai-mental-health-opportunities-and-challenges |
| Description | Symposium on Artificial Intelligence in Health & Care: Regulation, Evaluation & Policies |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Policymakers/politicians |
| Results and Impact | Artificial Intelligence has already proven its potential to reshape healthcare technologies. It offers improved diagnosis, intervention, health system efficiency, and preventive medicine in the hands of health professionals, patients, carers, and the public. However, established ways of testing and regulating technologies are not easy to apply to AI. In this symposium, presentations and panel discussion will explore: Evaluation and regulatory approaches, and how they can be developed to allow for faster and safer innovation. New horizons in AI, some of which will make it easier to ensure AI is explainable, reliable, while others will create new challenges for evaluation and regulations. The Symposium is for researchers and professionals in healthcare, government, business and academia. It will be run in three blocks running on the 7, 17 and 21 of September as a hybrid event on campus and online (Teams) starting at 14.30 each day. Speakers and panel members confirmed include experts from MHRA, NHSX, American College of Radiology, Roche Diagnostics, Kheiron, British Standards Institute, and universities and academic medical centres of Birmingham, Cambridge, Harvard, Leeds, Oxford, Stanford, Duke University, UCL and Imperial College London. |
| Year(s) Of Engagement Activity | 2021 |
| URL | https://www.eventbrite.co.uk/e/ai-in-future-healthcare-regulation-evaluation-policies-registration-1... |
| Description | World Economic Forum - AI Summit Spring 2024 and Fall 2024 |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Policymakers/politicians |
| Results and Impact | Building on the success of the inaugural Responsible AI Leadership Summit and the launch of the AI Governance Alliance (AIGA), the AI Governance Summit brings together more than 200 AI leaders, including Alliance community members and a broader AI community of experts and influential decision-makers. This collaborative gathering serves as a pivotal platform for knowledge sharing, strategic outlook and the formulation of concrete action plans to ensure the responsible development and deployment of generative AI on a global scale. I was an invited member. |
| Year(s) Of Engagement Activity | 2023,2024 |
| URL | https://www.weforum.org/events/ai-governance-summit-2023/ |