Developing Novel Machine Learning Techniques to Improve Comparative Judgements for e-Learning and e-Assessment

There is a wide selection of aims we could examine under this project:
1) To understand how machine learning (ML) can function as a support for an educator and not a strait jacket. Key questions to ask:
Are different measures relevant to different teachers? How do they interpret the data and use it to guide their teaching?
2) To understand how we can gather large volumes of quantifiable data about qualitative assessments allowing machine learning (ML) approaches to work on the data.
3) What design methods are suitable to use in order to engage with students and teachers in this complex design process. Key questions are:
How do we express possibilities for different insights that we can gather from the data?
How do we capture ideas about more nebulous elements of ML like privacy and ensure ongoing consent for data collection?
4) To identify appropriate ML methods and develop a complete ML framework for analysing qualitative assessments.
5) To actively query a small subset of all submissions to learn what attributes constitute excellent quality, taking an interactive approach with the educators to improve our proposed
framework's performance.
6) To validate and make the developed tool useable in the real-world.

We aim to redress the imbalance caused by automated marking tools promoting specific approaches to assessment. We will develop a decision support tool for educators assessing qualitative work that will make such assessments more attractive by increasing the educators understanding of the student cohorts' work and, potentially, reducing the amount of time they need to spend marking it. A comparative judgement framework [1] will be developed to allow educators to understand how changes in practice between different cohorts' impact on assessments. If the comparative judgement framework can show reliable performance at the cohort level, we will explore how it can be applied to individual qualitative assignments to support educators' assessments of them.
Year One - Identifying assessments: Qualitative assessments are subjective and learning about good and bad practices autonomously challenging. To address this, the student will survey teaching practitioners to identify the most important types of qualitative assessments, and key learning outcomes in those assessments. We will also work with educators and students to understand their attitudes to automated assessment of their work, their concerns about the tool and the ways in which they can be reassured of the validity of such assessments following Value Sensitive Design approaches [2]. This will narrow down the
scope of the tool and be used to create a proof-of-concept.
Year One and Two - Learning knowns and identifying gaps in knowledge: CDSM has a
wealth of data on submissions and their respective grades. At this stage, we aim to learn from this data to identify key features within the scope identified in the previous step. Here, we will first focus on finding meaningful structures in the dataset from a semantic perspective [3], potentially utilising a supervised approach (using existing grades) and some form of contextual embeddings [4]. These will help derive insight into what makes a submission address specific
learning outcomes. Given we use a probabilistic model, we can deduce which submissions our model has low confidence (or high uncertainty in predictions) about: this shows the gaps in our knowledge, which only a human practitioner can help fill.
Year Three - Interactively improving models: develop an interactive visual system to display insights into cohort data and individual submissions and allow educations practitioners to respond to two key questions about a submission:
Were our model predictions correct? How would they rate a given assignment?
These submissions will be carefully picked via an active learning strategy where we select submission that we are unsure about or that are likely to improve our model to provide better predictions.

The Centre will nurture 55 new PhD researchers who will be highly sought after in technology companies and application sectors where data and intelligence based systems are being developed and deployed. We expect that our graduates will be nationally in demand for two reasons: firstly, their training occurs in a vibrant and unique environment exposing them to challenging domains and contexts (that provide stretch, ambition and adventure to their projects and capabilities); and, secondly, because of the particular emphasis the Centre will put on people-first approaches. As one of the Google AI leads, Fei-Fei Li, recently put it, "We also want to make technology that makes humans' lives better, our world safer, our lives more productive and better. All this requires a layer of human-level communication and collaboration" [1]. We also expect substantial and attractive opportunities for the CDT's graduates to establish their careers in the Internet Coast region (Swansea Bay City Deal) and Wales. This demand will dovetail well with the lifetime of the Centre and provide momentum for its continuation after the initial EPSRC investment.

With the skills being honed in the Centre, the UK will gain a important competitive advantage which will be a strong talent based-pull, drawing in industrial investment to the UK as the recognition of and demand for human-centred interactions and collaborations with data and intelligence multiplies. Further, those graduates who wish to develop their careers in the academy will be a distinct and needed complement to the likely increased UK community of researchers in AI and big data, bringing both an ability to lead insights and innovation in core computer science (e.g., in HCI or formal methods) allied to talents to shape and challenge their research agenda through a lens that is human-centred and that involves cross-disciplinarity and co-creation.

The PhD training will be the responsibility of a team which includes research leaders in the application of big data and AI in important UK growth sectors - from health and well being to smart manufacturing - that will help the nation achieve a positive and productive economy. Our graduates will tackle impactful challenges during their training and be ready to contribute to nationally important areas from the moment they begin the next steps of their careers. Impact will be further embedded in the training programme with cohorts involved in projects that directly involve communities and stakeholders within our rich innovation ecology in Swansea and the Bay region who will co-create research and participate in deployments, trials and evaluations.

The Centre will also impact by providing evidence of and methods for integrating human-centred approaches within areas of computational science and engineering that have yet to fully exploit their value: for example, while process modelling and verification might seem much removed from the human interface, we will adapt and apply methods from human-computer interaction, one of our Centre's strengths, to develop research questions, prototyping apparatus and evaluations for such specialisms. These valuable new methodologies, embodied in our graduates, will impact on the processes adopted by a wide range of organisations we engage with and who our graduates join.

Finally, as our work is fully focused on putting the human first in big data and intelligent systems contexts, we expect to make a positive contribution to society's understandings of and involvement with these keystone technologies. We hope to reassure, encourage and empower our fellow citizens, and those globally, that in a world of "smart" technology, the most important ingredient is the human experience in all its smartness, glory, despair, joy and even mundanity.

[1] https://www.technologyreview.com/s/609060/put-humans-at-the-center-of-ai/