ROAD2H: Resource Optimisation, Argumentation, Decision Support and Knowledge Transfer to Create Value via Learning Health Systems

Lead Research Organisation: Imperial College London
Department Name: Surgery and Cancer

Abstract

ROAD2H will develop novel Learning Health System (LHS) techniques to facilitate uptake of health interventions to achieve Universal Health Coverage (UHC) in Low- and Middle-Income Countries (LMICs). UHC is a specific target in the UN Sustainable Development Goals (SDG 3.8), with many LMICs attempting to extend population coverage of quality healthcare via public-subsidised health insurance schemes (HISs). Two issues are key to achieving UHC via these schemes: i) how budgets should be best allocated among competing clinical interventions (allocative efficiency) ii) under what timing and circumstances particular interventions should be offered for what kinds of patients in which settings and to what standard of care (quality).Current attempts in LMICs to address efficiency and quality issues include Health Technology Assessment (HTA) based on cost-effectiveness analysis of individual healthcare interventions, and evidence-based guidelines/pathways recommending best practice within whole disease areas. However there have been limited uptake of system-wide mathematical optimisation techniques (in part due to significant data requirements); no efforts to using data routinely collected by HISs for billing and claims purposes, as well as Electronic Health Records (EHRs), for improving efficiency and quality, even though existing attempts among LMICs to use such data retrospectively as inputs into HTA and guidelines show promise.

We hypothesise that a LHS framework for national HISs, combining resource optimization, argumentation (as understood in AI), clinical decision support and clinical and policy analytics, will facilitate the uptake of evidence-based, cost-effective and data-driven healthcare interventions, and improve the overall efficiency and quality of integrated care systems in LMICs transitioning to UHC.

ROAD2H will deliver DSS tools that integrate individual patient data (drawn from EHRs) with localized clinical guidelines (including HTA, clinical pathways) obtained via policy and clinical analytics from best available clinical and cost-effectiveness evidence as well as HIS billing/claim data. We will use argumentation to parameterize optimisation problems drawn from cost-effectiveness and resource optimization considerations, and then use standard methods in mathematical optimisation to maximize efficiency of clinical interventions. We will use argumentation also to integrate optimization for maximal efficiency and reasoning with localized guidelines, and resolve conflicts as well as transparently explain recommendations for clinical interventions. The DSS will make use of these explained recommendations as well as data provenance techniques to generate and transparently collect patient-tailored recommendations to enable further learning, in line with the LHS methodology. The DSS tools will operate at the EHR/clinician/patient interaction and integrate data from multiple sources as well as have localised interfaces (by country and possibly hospital/clinician).

The project will be carried out in three phases:
- Phase 1: To review existing HISs and infrastructures in three exemplar countries (China, Serbia, Myanmar), against the requirements for LHSs
- Phase 2: Iterative development of a prototype digital methodology (encompassing policy, data, cost-effectiveness analytics, optimization, argumentation, DSS and provenance) to be embedded into existing HISs
- Phase 3: To evaluate, by means of proof-of-concept prototype DSS tools in Serbia and China and by scoping in Myanmar, the technical and clinical feasibility of the ROAD2H methodology, its acceptability among end users, and its potential and preconditions for generalization across disease areas and health systems settings in order to support UHC.

Planned Impact

As its primary purpose, ROAD2H has the potential to directly benefit 3 ODA countries, ranging low-income (Myanmar) to middle income (Serbia and China, with a focus on the latter's rural poor), through:
- reducing out of pocket healthcare spending and financial impoverishment,
- minimising households' precautionary saving thus boosting economic growth,
- reducing waste and improving efficiency of healthcare spending by governments and insurers, allowing fairer and more comprehensive coverage, contributing to the Sustainable Development Goal 3.8 on UHC.

As LMIC budgets grow, demands on those budgets grow too, both from expensive new healthcare technologies, an expanding base of citizens entitled to care through UHC and growing non-communicable disease burdens with growing wealth. Health ministries have limited capacity to control their investments or to make a case to treasuries for targeted investments in high priority diseases and populations. The resulting inefficiencies and inequities affect health as well as economic growth and political stability. ROAD2H will address these challenges by empowering payers with better information as to where their investment goes, and how such allocation compares to global but locally adapted norms for best practice. Our diverse sample of countries will enable us to adapt our methodologies to address the different challenges faced by LMICs: Serbia, a small European nation with a well-established HIS and billing system; China, a vast country with a relatively new HIS and a fragmented healthcare delivery model with low penetration of best practice; and Myanmar, one of the world's poorest countries, spending about 1% of its GDP on health but with major potential in leapfrogging developed economies through better use of digital technology.

There could be considerable impact from piloting ROAD2H methodologies among three regions of China (Xiamen City, Shaanxi, Henan) and Serbia, with a combined population of over 80m. There is also immense potential for scale-up across rural China (750m rural residents insured under the New China Medical Scheme), in addition for adapting and testing our methodologies in other ODA countries moving to UHC with nascent or established national health insurance schemes, where IGHI has strong links with senior policymakers (inc. India, Ghana, South Africa, Indonesia, Vietnam, Thailand) and where most of the world's poor reside. Our proposal is a much needed and timely contribution to developing countries' efforts towards the Sustainable Development Goal 3.8, aligning well with the strategic priorities of the Global Challenge Research Fund, namely 'sustainable health and wellbeing'.

Further, UN and multilateral organisations such as the World Bank recognise that Big Data will play an important role in monitoring countries' implementation of SDGs, however there currently exist few applications within health beyond infectious disease surveillance or m/eHealth platforms at the patient-clinician interface. We propose that to go much further by leveraging routinely collected data to optimise budget allocation and improve best practice at the system-wide level. This will help countries increase the efficiency and quality of their investment, improving patient referral and care integration across health system tiers, strengthen the generation and translation of locally-relevant knowledge, and ultimately realise UHC.

As a secondary objective, in addition to significant healthcare benefits in LMICs where the use cases will be developed and beyond, ROAD2H will ensure that the UK is at the forefront of the development of sophisticated knowledge and data management systems to advance healthcare provision and assist LMICs in enhancing their healthcare systems. ROAD2H will also strengthen foundations for UK-based commercial opportunities, expanding on our model of innovative public-private partnerships interfacing the healthcare, digital and creative sector
 
Description Designing a treatment path for a patient suffering from multiple conditions involves merging and applying multiple clinical guidelines and is recognised as a difficult task. This is
especially relevant in the treatment of patients with multiple chronic diseases, such as chronic obstructive pulmonary disease, because of the high risk of any treatment change having potentially lethal exacerbations. Clinical guidelines are typically designed to assist a clinician in treating a single condition with no general method for integrating them. Additionally, guidelines for different conditions may contain mutually conflicting recommendations with certain actions potentially leading to adverse effects. Finally, individual patient preferences need to be respected when making decisions.
In this work we present a description of an integrated framework and a system to execute conflicting clinical guideline recommendations by taking into account patient specific information and preferences of various parties. Overall, our framework combines a patient's electronic health record data with clinical guideline representation to obtain personalised recommendations, uses computational argumentation techniques to resolve conflicts among recommendations while respecting preferences of various parties involved, if any, and yields conflict-free recommendations that are inspectable and explainable. The system implementing our framework will allow for continuous learning by taking feedback from the decision makers and integrating it within its pipeline.

We have:
1. Successfully implemented argumentation as an extension to the transition-based medical recommendation (TMR) model
2. Implemented TMR in an electronic health system system to provide recommendations
3. Integrated optimisation and argumentation by simulating a model for real time analysis of a complex economic model.
Exploitation Route Future work on implementations of TMR involves constructing standard interfaces that can be queried ad-hoc by other systems such as argumentation frameworks. Similar efforts to increase the interoperability of EHR data include the examination of techniques to standardise independent vendor formats to a single target format.

Firstly, we will aim to account for various types of interactions that accompany the TMR model, including those concerning conflicts such as side-effects, but also other interactions such as safety. We will also study the integration of preferences from various sources, and possible interactions of those preferences. In addition, we will explore various ways of extracting explanations from the argumentative reasoning process, such as visualising arguments and their relationships, as well as using natural language generation to yield textual explanations of the reasoning outcomes.

Currently the TMR model work is being taken forward in several ways. 1. Prof Alessandra Russo is leading a proposal to UKRI 'AIDM: Intelligible AI for Decision Making' with BD as a Co-I, and with support from NICE and the UK Faculty of Clinical Informatics. The ROAD2H model is being used as a framework for linking general AI models (neuro-symbolic reasoning) with clinical context to provide both reasoning/explanation and reinforcement learning. 2. KCL/Prof Vasa Curcin are in discussion with both Heliant in Serbia and NICE in the UK about further demonstrator projects.

Finally, we will make use of the well-established theoretical properties of regarding reasoning and preferences, and establish what they mean in the context of medical decision making.
Sectors Digital/Communication/Information Technologies (including Software),Healthcare

 
Description The impact of AI on many economic sectors, including healthcare will be profound over the coming years. There are however a number of approaches to AI implementation using different methods of knowledge representation and different means of learning, these include neural networks deep learning models, ontologies and more recently Large Language Models. ROAD2H was based on the ontological principle, using an ontological framework to express the effects, evidence and direction of guideline recommendations, thus making them computable. We enabled argumentation and explanation via this approach, currently lacked by more 'black box' AI models. Increasingly such frameworks are being seen as a means to provide an explainability layer to powerful but non-explainable models, and to aid in reinforcement learning to train larger more general models for specific tasks. ROAD2H is thus a practical healthcare application with foundational implications in the field of AI.
First Year Of Impact 2022
Sector Healthcare
Impact Types Economic

 
Description Argumentation-based Deep Interactive eXplanations
Amount € 2,500,000 (EUR)
Funding ID 101020934 
Organisation European Research Council (ERC) 
Sector Public
Country Belgium
Start 10/2021 
End 09/2026
 
Description Developing and implementing Machine Learning driven analytics for quality improvement in healthcare
Amount £337,149 (GBP)
Funding ID 707135 
Organisation The Health Foundation 
Sector Charity/Non Profit
Country United Kingdom
Start 02/2018 
End 09/2021
 
Description London Substantive Site for HDR UK
Amount £6,000,000 (GBP)
Organisation Health Data Research UK 
Sector Private
Country United Kingdom
Start 04/2018 
End 03/2023
 
Description Collaboration with Clinical-Hospital Centre Dr DragiÅ¡a MiÅ¡ovic, Belgrade, Serbia 
Organisation University Hospital Center Dr Dragisa Misovic
Country Serbia 
Sector Hospitals 
PI Contribution Introduction the scope and methods of ROAD2H and use of decision support tools, mapping contributions by clinicians to generalised health semantics and clinical knowledge and decision making models.
Collaborator Contribution Exemplary patient scenarios, constraints, applied clinical guidelines, actual practices and challenges in the domain of CKD.
Impact Not yet. Nephrology with haemodialysis, research and education
Start Year 2017
 
Description Collaboration with Clinical-Hospital Centre Zvezdara, Belgrade, Serbia 
Organisation Zvezdara Medical Centre (KBC Zvezdara)
Country Serbia 
Sector Academic/University 
PI Contribution Introduction the scope and methods of ROAD2H and use of decision support tools, mapping contributions by clinicians to generalised health semantics and clinical knowledge and decision making models.
Collaborator Contribution Exemplary patient scenarios, constraints, applied clinical guidelines, actual practices and challenges in the domain of COPD and CKD. Expected direct collaboration with the Department of Pulmonology and Allergology with Immunology and Nephrology and Metabolic Disorders with Dialysis Department.
Impact Not yet. Pulmonology and nephrology, research and education with outreach to primary care.
Start Year 2017
 
Description Collaboration with Cornell Weill and Northwestern University on phenomics 
Organisation Cornell University
Department Weill Cornell Medicine
Country United States 
Sector Academic/University 
PI Contribution Dr Curcin leads the Informatics group at School of Population Health and Environmental Sciences and has recently been awarded an HDR UK Phenomics grant to develop the informatics infrastructure for a UK National Human Phenome Portal, having originally pioneered some of the informatics techniques behind this effort back in the 2000-s. Additionally, his group is active in developing phenotypes using Machine Learning from Twitter and Facebook. The collaboration between Cornell and KCL is timely and will strengthen the output of both groups and open the doors to further collaborations. We obtained seed funding from King's and Weill Cornell to cover the costs of two visits to Weill Cornell Medical by Dr Curcin and Dr Chapman, researcher working on the HDR UK Phenomics project. The first visit will involve introductions to the team and detailed discussions around the issues that KCL team will be facing in the next period, namely: a) the technology behind the computable phenotype representations in PheKB; b) disease areas that have been covered by PheKB in the past; c) adoption techniques employed to encourage take-up by US research teams. Curcin and Pathak will also consider options for joint grant applications and decide on the first one to work on. The second visit will review the PheKB/UK Phenome progress on both sides, and ensure standardisation work is on track. By this stage, UK Phenome repository will have some exemplar phenotypes, which we will try to port to PheKB. Similarly, we shall take a selection of relevant (stroke) phenotypes from PheKB and port them into UK Phenome database.
Collaborator Contribution Prof Jyotishman Pathak's group at Weill Cornell Medical School, together with the collaborators at Northwestern University is at the forefront of the phenomics research in the US, leading the PheKB phenotype database work and developing novel Machine Learning technologies for investigating heterogeneous phenotypes from EHRs and patient-produced data such as social media.
Impact Submission to the AMIA 2020 conference around PhenoFlow architecture for phenotype design.
Start Year 2019
 
Description Collaboration with Cornell Weill and Northwestern University on phenomics 
Organisation Northwestern University
Country United States 
Sector Academic/University 
PI Contribution Dr Curcin leads the Informatics group at School of Population Health and Environmental Sciences and has recently been awarded an HDR UK Phenomics grant to develop the informatics infrastructure for a UK National Human Phenome Portal, having originally pioneered some of the informatics techniques behind this effort back in the 2000-s. Additionally, his group is active in developing phenotypes using Machine Learning from Twitter and Facebook. The collaboration between Cornell and KCL is timely and will strengthen the output of both groups and open the doors to further collaborations. We obtained seed funding from King's and Weill Cornell to cover the costs of two visits to Weill Cornell Medical by Dr Curcin and Dr Chapman, researcher working on the HDR UK Phenomics project. The first visit will involve introductions to the team and detailed discussions around the issues that KCL team will be facing in the next period, namely: a) the technology behind the computable phenotype representations in PheKB; b) disease areas that have been covered by PheKB in the past; c) adoption techniques employed to encourage take-up by US research teams. Curcin and Pathak will also consider options for joint grant applications and decide on the first one to work on. The second visit will review the PheKB/UK Phenome progress on both sides, and ensure standardisation work is on track. By this stage, UK Phenome repository will have some exemplar phenotypes, which we will try to port to PheKB. Similarly, we shall take a selection of relevant (stroke) phenotypes from PheKB and port them into UK Phenome database.
Collaborator Contribution Prof Jyotishman Pathak's group at Weill Cornell Medical School, together with the collaborators at Northwestern University is at the forefront of the phenomics research in the US, leading the PheKB phenotype database work and developing novel Machine Learning technologies for investigating heterogeneous phenotypes from EHRs and patient-produced data such as social media.
Impact Submission to the AMIA 2020 conference around PhenoFlow architecture for phenotype design.
Start Year 2019
 
Description Collaboration with Dr Tiago Oliveira 
Organisation National Institute of Informatics (NII)
Country Japan 
Sector Public 
PI Contribution Kristijonas Cyras has personally collaborated with Dr Tiago Oliveira (National Institute of Informatics, at the time of collaboration) in developping research methodology, publishing papers and organising project-related events.
Collaborator Contribution Co-developping research methodology (on using argumentation for clinical reasoning) and co-authoring conference papers. Co-organising MedRACER workshop at KR 2018 conference.
Impact Multidisciplinary collaboration: artificial intelligence and clinical informatics. Conference paper: K. Cyras, T. Oliveira: Argumentation for Reasoning with Conflicting Clinical Guidelines and Preferences. The 16th International Conference on Principles of Knowledge Representation and Reasoning (KR 2018). Conference paper: K. Cyras, T. Oliveira: Resolving Conflicts in Clinical Guidelines using Argumentation. International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2019). [DOI as yet unavailable] Research event: MedRACER workshop at KR 2018. https://sites.google.com/view/medracer .
Start Year 2018
 
Description Collaboration with Imperial College on use of template-based provenance for Learning Health System applications 
Organisation Imperial College London
Country United Kingdom 
Sector Academic/University 
PI Contribution The provenance template technology developed by myself and my team in ProvTemp is being used as the basis for the provenance infrastructure in Learning Health System decision support research at Imperial's Institute of Global Health Innovation.
Collaborator Contribution The partners are providing the medical use cases that could benefit from the provenance infrastructure.
Impact Successfully funded projects - EPSRC ROAD2H and CRUK Demonstrating the feasibility of a Learning Health System for cancer diagnosis in primary care. Through the former, our decision support infrastructure, including the provenance module will be deployed in health systems in Serbia and China. This is a multi-discipinary collaboration spanning improvement science, medicine, computer science, and medical informatics.
Start Year 2017
 
Description Collaboration with Masaryk University, Czech Republic on non-repudiation of provenance 
Organisation Masaryk University
Country Czech Republic 
Sector Academic/University 
PI Contribution Through King's work, provenance templates have by now become a recognised methodology for the construction of data provenance records. Each template defines the provenance of a domain-specific action in abstract form, which may then be instantiated as required by a single service call to the provenance template server. As data reliability and trustworthiness becomes a critical issue in an increasing number of domains, there is a corresponding need to ensure that the provenance of that data is non-repudiable. This can be achieved by using a third-party trusted notary to store supporting evidence of the data provenance being recorded. In collaboration with Masaryk Unviersity, we developed two new, complementary modules to our template model and implementation to produce non-repudiable data provenance. The first, a module that traces the operation of the provenance server itself, and records a provenance trace of the construction of an object-level document, at the level of individual service calls. The second, a non-repudiation module that generates evidence for the data recorded about each call, annotates the server trace accordingly, and submits a representation of that evidence to a provider-agnostic notary service. We also contributed use cases developed from the decision support projects we are working on, namely EPSRC CONSULT and ROAD2H.
Collaborator Contribution Masaryk University contributed their expertise in the developing ISO TC 276 standard and previous non-repudiation work (outside of the provenance template domain).
Impact The main output so far is a paper that will be submitted to Provenance Week 2020, and an expanded version to FGCS journal. The paper evaluates the applicability of our approach in the context of a clinical decision support system (e.g. EPSRC CONSULT and ROAD2H). We first demonstrate the suitability of our solution with respect to a security threat model for such an environment. We then select three use cases from within the system with contrasting data provenance recording requirements and analyse the subsequent performance of our prototype implementation against three different notary providers. The work will also feed in into the work ISO TC276 WP6 working group that Curcin, Fairweather, Holub, and Wittner are all members of.
Start Year 2020
 
Description Collaboration with National Institute for Health and Care Excellence on use of provenance for managing recommendations and evidence 
Organisation National Institute for Health and Care Excellence (NICE)
Country United Kingdom 
Sector Public 
PI Contribution My team is applying the Provenance Template modeling method to the challenges that NICE has in managing their research metadata. Specifically, they are interested in exploring the versioning of their guideline recommendations, and its relationship to the changing evidence base.
Collaborator Contribution NICE is conducting a survey of their stakeholders (industrial partners, clinical organizations etc.) to understand their needs with respect to data provenance of NICE guidelines. This is a valuable piece of work that they are uniquely positioned to deliver, and will be of significant use to my team in developing the provenance research portfolio further. As a secondary benefit, the NICE employee placed in my group is advising on the ROAD2H and CONSULT projects which both have elements of guideline modeling, to ensure its applicability to the UK national requirements.
Impact We have jointly obtained the "Towards Computable Guidelines", MRC Industry Proximity Award funding worth £30K, to establish a pilot collaboration through hosting a NICE employee in my group, and are currently working on a paper and a larger grant proposal. This is a multi-disciplinary collaboration that spans informatics and public health.
Start Year 2017
 
Description Collaboration with University of Belgrade and Heliant Ltd. 
Organisation University of Belgrade
Country Serbia 
Sector Academic/University 
PI Contribution In collaboration with University of Belgrade and Heliant Ltd. King's is exploring the opportunities for deploying its Learning Health System technologies in Serbia. So far this has been focused on decision support systems, but we are now looking into making Serbian health data sets research ready through joining the OHDSI's OMOP partnership.
Collaborator Contribution Access to data and hospitals in Serbia, developer resources.
Impact Joint papers in JMIR Medical Informatics and AMIA conferences.
Start Year 2016
 
Description Collaboration with University of Birmingham on ROAD2H computable guidelines 
Organisation University of Birmingham
Country United Kingdom 
Sector Academic/University 
PI Contribution The King's team is contributing its reasoning engine for computable guidelines, developed originally within the ROAD2H project, to a joint project on investigating combined guidelines for common multimorbidities.
Collaborator Contribution The partners are bringing in the extensive track record of assembling computable guidelines in ProForma, and working with the Dexter tool for automating epidemiological analysis.
Impact Draft ideas for EPSRC and MRC grants so far.
Start Year 2023
 
Description Collaboration with Weill Cornell Medicine in NYC, USA around Learning Health Systems and Cancer 
Organisation Cornell University
Department Weill Cornell Medicine
Country United States 
Sector Academic/University 
PI Contribution Prof Rulla Tamimi is head of cancer epidemiology at the Meyer Cancer Centre, WCM and Prof Mark Weiner is Clinical Director of INSIGHT, the New York City Clinical Data Research Network. We are jointly working together on a funding proposal to the Moore Foundation diagnostic excellence theme with a view to establishing a learning system to support cancer diagnosis in The Bronx and Brooklyn.
Collaborator Contribution We have been working together over the past year to present an idea to the MF and now to develop a grant proposal for consideration in 2021.
Impact Delaney BC. Cancer and Learning Health Systems, potential in New York City. Divisional seminar, Division of Population Health Sciences, Weill Cornell Medicine, New York USA, 3rd Feb 2020.
Start Year 2019
 
Description Collaboration with the National Health Insurance Fund (NHIF) of Serbia 
Organisation National Health Insurance Fund
Country Serbia 
Sector Charity/Non Profit 
PI Contribution Bridging between the research context of ROAD2H and practical operational context of the health financing and the Serbian healthcare.
Collaborator Contribution Expected feedback on needs in terms of knowledge extraction, adherence, trends analysis, policy and guidance development, form the perspective of the financial management of health system. Possible provisioning of depersonalised data for ROAD2H. Feedback on applicability of project's outputs.
Impact None yet.
Start Year 2017
 
Title Bipolar Argumentation Solver 
Description Bipolar Argumentation Solver is both a stand-alone system and a web application implementing computations with the argumentation formalism Bipolar ABA. It has applications to solving reasoning tasks in fields such as AI, Law and Medicine, for instance representing and resolving a debate. 
Type Of Technology Webtool/Application 
Year Produced 2019 
Open Source License? Yes  
Impact This tool was presented at a major Artificial Intelligence conference (AAMAS) and has served as a backbone for further developing argumentation solvers that automate reasoning in the medical domain. 
URL https://bipolar-aba-solver.netlify.com/
 
Title Schedule Explainer 
Description This stand-alone software tool is used to explain makespan schedules (Mathematical Optimisation) using abstract argumentation frameworks (Artificial Intelligence). It provides the user with explanations regarding the goodness of schedules in makespan scheduling problem instances thus helping the user to better understand the problem, the optimisation solver and the solution. 
Type Of Technology Software 
Year Produced 2019 
Open Source License? Yes  
Impact This tool was used as a backend for developing an application that enables explainable scheduling for nurse rostering. It was also presented to various audiences of researchers and inspired further research into argumentation-supported explainable scheduling. 
 
Title Schedule Explainer for Nurse Rostering 
Description A proof-of-concept application for explaining nurse rostering. The tool illustrates how argumentation-supported explainable scheduling can be applied to real-world scenaria such as nurse rostering in clinical settings. 
Type Of Technology Webtool/Application 
Year Produced 2019 
Open Source License? Yes  
Impact This application has been used to engage numerous general as well as specialist audiences. It was presented to the general public during The Great Exhibition Road Festival and participated in British Computer Society (BCS) Specialist Group on AI (SGAI) Machine Intelligence Competition in 2019. 
URL https://youtu.be/pVXj_whrTkA
 
Description AI for Complex Clinical Decision Support at The Forum: Workshop on Artificial Intelligence and Health 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Policymakers/politicians
Results and Impact Kristijonas Cyras gave a talk on Artificial Intelligence for Complex Clinical Decision Support at Imperial's policymaker engagement event (The Forum). Attendees included, for instance, representatives from the Cabinet Office and from Innovate UK. The presentation(s) were foloowed by discussions on the advantages and challenges in deploying artificial intelligence to enhance healthcare in the UK.
Year(s) Of Engagement Activity 2019
 
Description Appearance in Imperial's enterprise industry issue (post Explain AI@Imperial workshop) 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact After Explain AI@Imperial workshop (http://www3.imperial.ac.uk/newsandeventspggrp/imperialcollege/engineering/computing/eventssummary/event_3-4-2018-11-47-39), Kristijonas Cyras has been asked to comment on his research concerning explainable artificial intelligence supporting medical decision making, for engagement purposes. Kristijonas then gave an interview which was incorporated into an (Imperial's) industry issue.
Year(s) Of Engagement Activity 2018
URL https://www.imperial.ac.uk/enterprise/issues/explainable-ai/
 
Description Argumentation-based clinical decision support system in ROAD2H 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Keynote talk by Francesca Toni at MedRACER 2018
Year(s) Of Engagement Activity 2018
URL https://sites.google.com/view/medracer/programme
 
Description Argumentation-based clinical decision support systems in ROAD2H 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Invited talk at Computational Medicine Conference, Imperial College London, March 2018
Year(s) Of Engagement Activity 2018
 
Description BCS SGAI Machine Intelligence Competition 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Interactive Schedule Explainer, an intelligent system applied to explainable nurse rostering, shortlisted for and presented at the final of British Computer Society's Machine Intelligence Competition. Imperial College London, Department of Computing researchers took part at the competition and showcased the proof-of-concept argumentation-supported explainable scheduling application to an audience of researchers and industry representatives alike.
Year(s) Of Engagement Activity 2019
URL http://www.road2h.org/news/kristijonas-%C4%8Dyras-amin-karamlou-and-team-bcs-sgai-machine-intelligen...
 
Description Bridging Clinical Workshop, National Academy of Sciences, Washington DC, USA 4-5th March 2019: The ROAD2Health Initiative and TRANSFoRm Project 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Industry/Business
Results and Impact 2-day workshop/conference at NAS. Mixture of academics, industry, tech startups and regulators (FDA).
discussion about using the LHS concept to drive research and care.
Presented progress with the ROAD2H project.
Year(s) Of Engagement Activity 2019
URL https://www.bridgingclinical.com/2019-agenda/
 
Description Decision support in ROAD2H 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Invited talk by francesca Toni at the People Like You networking Workshop, 15 January 2019
Year(s) Of Engagement Activity 2019
 
Description Departmental seminar in Edinburgh 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Postgraduate students
Results and Impact Using data standards and Ontologies to create a Learning Health System. The Farr Institute Scotland, Edinburgh, Sept 8th 2017.
Year(s) Of Engagement Activity 2018
 
Description Explainability Expert Roundtable at The Turing 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact The Alan Turing Institute organised an explainability expert roundtable in 2019 March 13 involving leading UK-based Explainable AI (XAI) researchers to discuss the state-of-the-art of AI explainability in research and real-life. The discussion was supposed to inform the Turing's Explainable AI guidelines meant for policy makers, industry and general public.
Year(s) Of Engagement Activity 2019
 
Description From data and rules to argumentation frameworks 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Professional Practitioners
Results and Impact Invited seminar at University of Swansea, 11 December 2018
Year(s) Of Engagement Activity 2018
 
Description From data and rules to argumentation frameworks 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Invited Talk by Francesca Toni at COMMA 2018, Warsaw, Poland. Included overview of ROAD2H
Year(s) Of Engagement Activity 2018
URL http://comma2018.argdiap.pl/conference-schedule/
 
Description Imperial Fringe Festival 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact Imperial Fringe 2018 was themed "Intelligence Redesigned", celebrating the latest developments and applications in artificial intelligence by Imperial researchers to solve some of the great challenges of our time. The Department of Computing represented the project, discussing applications of argumentation and optimisation in healthcare (the ROAD2H project) alongside other applications, and generated enthusiastic interest among the audiences. One direct outcome was that a junior doctor with an interest in technology in healthcare has made contact with ROAD2H team to request participation in the work.
Year(s) Of Engagement Activity 2011,2012,2013,2014,2015,2016,2017,2018
URL https://www.imperial.ac.uk/festival/fringe/
 
Description K Cyras's talk at Swansea University 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Professional Practitioners
Results and Impact Kristijonas Cyras gave an invited talk on Argumentation-enabled Explainable AI Applications at the University of Swansea (Theory Group seminar, Department of Computer Science) on 2019 March 21.
Year(s) Of Engagement Activity 2019
 
Description Legal, Ethical and Social Implications of Learning Systems for Diagnosis. Ann Arbor, Michigan, USA, 8th-9th Nov 2018. 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Legal, Ethical and Social Implications of Learning Systems for Diagnosis. Ann Arbor, Michigan, USA, 8th-9th Nov 2018.

Wide participation of patient groups as speakers and audience.
Year(s) Of Engagement Activity 2018
 
Description MEDRACER18 workshop 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact The talks and papers in MEDRACER focussed mainly on the representation of, and reasoning with, clinical guidelines. Martin Chapman was invited as the representtive of the CONSULT project, and spoke about the work done so far on deriving treatment plans from abstract guidelines, which involves using argumentation to resolve any contradictions within those guidelines based on patient preferences.
Year(s) Of Engagement Activity 2018
URL https://sites.google.com/view/medracer
 
Description Meeting at University of Michigan, Ann Arbor, to set the Learning Health System agenda for decision support 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Funded by the Gordon and Betty Moore Foundation to envision a Learning Health System for diagnostic excellence, University of Michigan held a one-day expert meeting to design and propose in more detail socio-technical infrastructure components for an LHS focused on diagnosis. Vasa Curcin was invited as the UK/European representative to the workshop, bringing in the experiences from CONSULT and ROAD2H.
Year(s) Of Engagement Activity 2019
 
Description Participating at The Great Exhibition Road Festival 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact Imperial College London, Department of Computing researchers led by Kristijonas Cyras took part at The Great Exhibition Road Festival (2019 June 29-30) presenting the AI-assisted Schedule Explainer tool for explaining nurse rostering in healthcare clinics. This was a showcase of applications of AI, particularly argumentation, to real-world optimisation problems, specifically scheduling in clinical settings. Thousands of people attended the Festival, hundreds saw the stand and over a hundred got engaged in the showcase.
Year(s) Of Engagement Activity 2019
URL http://www.road2h.org/news/kristijonas-%C4%8Dyras-amin-karamlou-and-team-great-exhibition-road-festi...
 
Description ROAD2H scoping / clinical use cases workshop in Xiamen, China with clinicians, IT staff and vendors 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Professional Practitioners
Results and Impact In January 2018, the ROAD2H held a two-day workshop in Xiamen, China, with clinicians, IT technical staff and EHR vendors from all three China pilot sites of the project (Xiamen, Ningqiang, and Wushan). At the meeting a shortlist of clinical priorities in COPD and CKD was discussed and input received as to what the local priorities amongst this list would be. The aim was to choose appropriate exemplars for the clinical use cases, that would be both technically and practically feasible and would lend themselves for evaluation. We also considered the setting in which the DSS would operate (village clinic, township clinic, outpatient, inpatient, high dependency, dialysis unit). Following the visit, a report was produced and the clinical decision problems were finalised, with upcoming activities for 2018 decided (fleshing out clinical use cases and parameters, and exploratory argumentation modelling)
Year(s) Of Engagement Activity 2018
 
Description ROAD2H website 
Form Of Engagement Activity Engagement focused website, blog or social media channel
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact The ROAD2H website was created primarily to facilitate outreach activities with potential partners in the wider network of the project (e.g. IT vendors and policymakers). One of the resources on the website is the ROAD2H Explainer, a two-page document explaining the key concepts and objectives of ROAD2H, and benefits for different groups of stakeholders; this document has also been translated into Serbian and Chinese.
Year(s) Of Engagement Activity 2017,2018
URL http://www.road2h.org
 
Description ROAD2H/CONSULT projects scoping workshop 
Form Of Engagement Activity Participation in an open day or visit at my research institution
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Professional Practitioners
Results and Impact ROAD2H researchers have invited colleagues from King's College London working on CONSULT project (https://gow.epsrc.ukri.org/NGBOViewGrant.aspx?GrantRef=EP/P010105/1) to scope space for collaboration regarding (but not limited to) the artificial intelligence methodologies used within the projects. The attendees exchanged formal presentations and discussed extensively the commoncalities and differences of the projects.
Year(s) Of Engagement Activity 2019
 
Description Reasoning with Ambiguous and Conflicting Evidence and Recommendations in Medicine (MedRACER 2018) 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact International workshop co-located with the 16th International Conference on Principles of Knowledge Representation and Reasoning (KR 2018), Tempe, Arizona, USA, October 29th, 2018
Year(s) Of Engagement Activity 2018
URL https://sites.google.com/view/medracer/home
 
Description Royal Society of Medicine talk 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact A Learning Health System for Diagnosis. 'Recent developments in Digital Health 2018'. Royal Society of Medicine, London Feb 27th 2018.
Year(s) Of Engagement Activity 2018
 
Description Satellite symposium at a conference 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Delaney BC. Using Real World Data to Translate Research into Practice: Realizing the Benefits of Learning Health Systems In: Pfizer Satellite Symposium at European Society for Medical Oncology, 26-28th September 2019, Barcelona, Spain. Real World Data in Oncology: its Growing Role in Research and Patient Care [Satellite symposium speaker]
Year(s) Of Engagement Activity 2019
 
Description Seminar at Carnegie Mellon 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact 80 postgraduate students and their supervisors attended a seminar given by Ruth Misener, which led to discussions on the topic and interest in future research
Year(s) Of Engagement Activity 2019
URL https://twitter.com/crislopeslara/status/1102990243752542208
 
Description Senior delegation from Chinese health policymakers (China National Health Development Research Center and National Health and Family Planning Commission) to Imperial College London 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Policymakers/politicians
Results and Impact Global Health and Development Group (GHD, formerly NICE International) has a deep and longstanding partnership with the China National Health Development Research Center (CNHDRC). Since 2015, GHD has hosted an annual study delegation of senior Chinese health bureaucrats in the UK to learn from the experiences of and developments in the UK National Health Service. In the visit to Imperial in November 2017, and catering to the expressed interest of the Chinese delegation in the application of innovative technology including AI and Big Data in health, the ROAD2H team at Imperial (GHD, Institute of Global Health Innovation, and Department of Computing) together with CNHDRC presented on the progress and planned approaches of ROAD2H in China. This resulted in the decision to hold the stakeholder engagement workshop in Jan 2018.
Year(s) Of Engagement Activity 2015,2016,2017
 
Description Symposium 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Professional Practitioners
Results and Impact Delaney BC. Artificial Intelligence and GP diagnosis. Optum Bazalgette Symposium, Maidstone Kent. 4th April 2019. [Keynote speaker]
25 invited guests to discuss the topic.
Year(s) Of Engagement Activity 2019
 
Description Symposium on data and health policy in China 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Policymakers/politicians
Results and Impact Delaney BC, Using data to inform real-world policy in global health systems. Workshop, Wuxi, China 14th August 2019. [International training workshop seminar]
Year(s) Of Engagement Activity 2019
 
Description The problem of supporting decision-making 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Invited talk by francesca Toni at the Workshop on Artificial Intelligence and the Learning Health System London, June 14th and 15th, 2018
Year(s) Of Engagement Activity 2018
 
Description Workshop 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Policymakers/politicians
Results and Impact Westminster Health Forum policy conference
The future for NICE in health and social care - new evaluation processes, keeping pace with
developments, and improving patient access to innovation
Timing: Morning, Monday, 25th January 2021

Prof Brendan Delaney gave a talk about the ROAD2H project
Year(s) Of Engagement Activity 2021
URL https://www.westminsterforumprojects.co.uk/conferences/westminster-health-forum
 
Description Workshop on AI and Diagnosis 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact Delaney BC, Kostopoulou O. Challenges in developing a Learning System for GP Diagnosis. Cancer Research UK, CanTest international graduate school. Jesus College Cambridge. 9th April 2019. [Keynote speaker]
Year(s) Of Engagement Activity 2019
 
Description Workshop on regulation of computable knowledge in healthcare 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Policymakers/politicians
Results and Impact Workshop held at the British Computer Society London. Discussion and presentations from NICE, MHRA, Royal Colleges (including Faculty of Clinical Informatics) and academia.
Year(s) Of Engagement Activity 2023
URL https://facultyofclinicalinformatics.co.uk/uk-mcbk
 
Description Workshop paper Argumentation for Explainable Reasoning with Conflicting Medical Recommendations at MedRACER'18 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Kristijonas Cyras has presented a research paper describing methodologies within ROAD2H to fellow researchers.
Year(s) Of Engagement Activity 2018
URL http://ceur-ws.org/Vol-2237/medracer-paper-2.pdf
 
Description Workshop paper Explanatory Predictions with Artificial Neural Networks and Argumentation at XAI (IJCAI 2018) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Kristijonas Cyras presented the paper Explanatory Predictions with Artificial Neural Networks and Argumentation at the second Explainable Artificial Intelligence (XAI) workshop held at IJCAI/ECAI 2018 conference (http://home.earthlink.net/~dwaha/research/meetings/faim18-xai/).
Year(s) Of Engagement Activity 2018
URL https://www.doc.ic.ac.uk/~kc2813/materials/papers/XAI2018_ANNA.pdf