Learning NLP Tasks with Multimodal Interactions

This project aims to develop a methodology that enables users to alter the behaviour of machine learning models using a combination of natural language instructions, dialogue, and other forms of feedback such as labels. The aim is for users to correct errors, provide missing information, and adapt models to new tasks and domains. Specifically, the project focuses on deep neural networks for multi-label text classification. The methodology will combine recent advances in contextualised embeddings with Bayesian approaches to actively request user inputs and handle the uncertainty in interpreting different forms of feedback. The project will actively seek out real-world use cases and incorporate user studies to motivate the design and evaluation of interactive learning methods.

The core objective is to develop a methodology that allows users to adapt and correct errors in deep neural networks using multiple sources of feedback, including natural language inputs. This project aims to carry out foundational research into interactive machine learning.

The project is currently undertaking one of the identified case-studies in collaboration with one of the supervisor's existing partners - Dr. Yvette Payne. This case study specifically involves the process of automatically annotating GP transcripts. With the overall intention of the study being the creation of a system that can automatically create and annotate a concise set of consultation notes when provided an audio recording of the consultation, it is our intention to expand beyond this by developing a system that can provide real-time diagnoses along with suggested further lines of inquiry.

It is our intention to create a novel methodology based off the concept of active learning. However, where active learning queries the user for additional labelled data, our methodology will instead query the user for additional information. Using the case study as an example, the system will provide the GP with a question to ask the patient that will yield a lower uncertainty score for the suggested diagnosis. The current intention is to train the system using a dataset containing transcribed GP consultations. [1]

While the methodology is initially being developed within the domain of healthcare and GP consultations, it can easily be applied to any context where a consultation is taking place between a professional and someone unfamiliar with that domain. All that requires is a change in the training material.

Through the successful development of this methodology within the identified case study along with any required ethical reviews, it may be possible to deploy such a system for use within the NHS, allowing for the standardisation of all GP consultation notes.

This project falls within the EPSRC Information and Communication Technologies (ICT), and Healthcare Technologies themes. As well as this, it falls under the wider research area of Artificial Intelligence and Robotics.

[1] Jepson, M., Salisbury, C., Ridd, M., Metcalfe, C., Garside, L. and Barnes, R., 2017. The 'One in a Million' study: creating a database of UK primary care consultations. British Journal of General Practice, 67(658), pp.e345-e351.