A Framework for Evaluating Diagnostic Imaging Artificial Intelligence Medical Tools

The project intended aims and Objectives
1) A working methodology for the design, evaluation & introduction of AI into existing Radiology workflow
2) Design & clinical evaluation toolkit for clinical staff
3) Good practice to be used during design that promotes trust in the product
4) Pre-procurement/deployment evaluation toolkit
5) Good practice guide to purchasing AI systems
6) Post procurement/deployment toolkit
7) Good practice guide to deploying an AI system into an existing workflow - engagement framework

For the Academy:
1) Research papers involving NIAW based authors
2) Enhanced toolkit for evaluation, service implementation and education

Practically:
Minimum of two sites for workshop & co-design, C&V and CTM UHBs, centred on a couple of Radiologists but this can expand depending upon the engagement as we discussed.
Other sites may wish to be involved and that would be a joint engagement exercise as we discussed (other Health Boards and Trusts). Other Groups of Radiology Reporter involvement could be considered and will enhance the project (Reporting Radiographers, Radiology Trainees) NHS X involvement - this would be beneficial, and involvement and/ or collaboration could be discussed.

The PhD timeline
In years 1 and 2, the student will need to work with the various stakeholders to organise co-design workshops and other forms of elicitation studies to inform planning for what needs
to be considered in such frameworks. This will require focusing on specific case-studies, and then working in a generative fashion to identify commonalities in which a common framework can be built upon. The specific methodologies employed will be decided by the student in partnership with the supervision team and PhD partner, responding to the student's understanding and the specific information needs they identify. A number have been proposed as part of this project (e.g., ethnographies, interviews, workshops, design fictions) and it is likely a mixture of these will be appropriate. During the second year, the student should be able to present multiple iterations of draft frameworks, while continuing to employ human-centred approaches to evoking reflections and guidance from stakeholders to further develop their framework, as well as developing guidance for how it can be used. This work may take the form of interactive co-design sessions where the draft framework is put into practice (e.g. with existing technologies or prototypes developed by the student). Through this the developed framework will evolve. PhD contributions In addition to producing the framework and guidance for its practical use-something that is lacking in the UK currently-the student will produce academic publications documenting their research practice for publication. The deliverables of the project will be of significant value for the stakeholder, and the academic and practical outcomes of the project hold the potential to shape procurement practice in many healthcare contexts in and outside of Wales.

The Centre will nurture 55 new PhD researchers who will be highly sought after in technology companies and application sectors where data and intelligence based systems are being developed and deployed. We expect that our graduates will be nationally in demand for two reasons: firstly, their training occurs in a vibrant and unique environment exposing them to challenging domains and contexts (that provide stretch, ambition and adventure to their projects and capabilities); and, secondly, because of the particular emphasis the Centre will put on people-first approaches. As one of the Google AI leads, Fei-Fei Li, recently put it, "We also want to make technology that makes humans' lives better, our world safer, our lives more productive and better. All this requires a layer of human-level communication and collaboration" [1]. We also expect substantial and attractive opportunities for the CDT's graduates to establish their careers in the Internet Coast region (Swansea Bay City Deal) and Wales. This demand will dovetail well with the lifetime of the Centre and provide momentum for its continuation after the initial EPSRC investment.

With the skills being honed in the Centre, the UK will gain a important competitive advantage which will be a strong talent based-pull, drawing in industrial investment to the UK as the recognition of and demand for human-centred interactions and collaborations with data and intelligence multiplies. Further, those graduates who wish to develop their careers in the academy will be a distinct and needed complement to the likely increased UK community of researchers in AI and big data, bringing both an ability to lead insights and innovation in core computer science (e.g., in HCI or formal methods) allied to talents to shape and challenge their research agenda through a lens that is human-centred and that involves cross-disciplinarity and co-creation.

The PhD training will be the responsibility of a team which includes research leaders in the application of big data and AI in important UK growth sectors - from health and well being to smart manufacturing - that will help the nation achieve a positive and productive economy. Our graduates will tackle impactful challenges during their training and be ready to contribute to nationally important areas from the moment they begin the next steps of their careers. Impact will be further embedded in the training programme with cohorts involved in projects that directly involve communities and stakeholders within our rich innovation ecology in Swansea and the Bay region who will co-create research and participate in deployments, trials and evaluations.

The Centre will also impact by providing evidence of and methods for integrating human-centred approaches within areas of computational science and engineering that have yet to fully exploit their value: for example, while process modelling and verification might seem much removed from the human interface, we will adapt and apply methods from human-computer interaction, one of our Centre's strengths, to develop research questions, prototyping apparatus and evaluations for such specialisms. These valuable new methodologies, embodied in our graduates, will impact on the processes adopted by a wide range of organisations we engage with and who our graduates join.

Finally, as our work is fully focused on putting the human first in big data and intelligent systems contexts, we expect to make a positive contribution to society's understandings of and involvement with these keystone technologies. We hope to reassure, encourage and empower our fellow citizens, and those globally, that in a world of "smart" technology, the most important ingredient is the human experience in all its smartness, glory, despair, joy and even mundanity.

[1] https://www.technologyreview.com/s/609060/put-humans-at-the-center-of-ai/