Turing AI Fellowship: Interactive Annotations in AI
Lead Research Organisation:
University of Bristol
Department Name: Engineering Mathematics and Technology
Abstract
With the prevalence of data-hungry deep learning approaches in Artificial Intelligent (AI) as the de facto standard, now more than ever there is a need for labelled data. However, while there have been interesting recent discussions on the definition of readiness levels of data, the same type of scrutiny on annotations is still missing in general: we do not know how or when the annotations were collected or what their inherent biases are. Additionally, there are now forms of annotation beyond standard static sets of labels that call for a formalisation and redefinition of the annotation concept (e.g., rewards in reinforcement learning or directed links in causality).
During this Fellowship we will design and establish the protocols for transparent annotations that empowers the data curator to report on the process, the practitioner to automatically evaluate the value of annotations and the users to provide the most informative and actionable feedback. This Fellowship will address all these through a holistic human-centric research agenda, bridging gaps in fundamental research and public engagement with AI.
The Fellowship aims to lay the foundations for a two-way approach to annotations, where the paradigm is shifted from annotations simply being a resource to them becoming a means for AI systems and humans to interact. The bigger picture is that, with annotations seen as an interface between both entities, we will be in a much better position to guide the relation of trust in between learning systems and users, where users translate their preferences into the learning systems' objective functions. This approach will help produce a much needed transformation in how potentially sensitive aspects of AI become a step closer to being reliable and trustworthy.
During this Fellowship we will design and establish the protocols for transparent annotations that empowers the data curator to report on the process, the practitioner to automatically evaluate the value of annotations and the users to provide the most informative and actionable feedback. This Fellowship will address all these through a holistic human-centric research agenda, bridging gaps in fundamental research and public engagement with AI.
The Fellowship aims to lay the foundations for a two-way approach to annotations, where the paradigm is shifted from annotations simply being a resource to them becoming a means for AI systems and humans to interact. The bigger picture is that, with annotations seen as an interface between both entities, we will be in a much better position to guide the relation of trust in between learning systems and users, where users translate their preferences into the learning systems' objective functions. This approach will help produce a much needed transformation in how potentially sensitive aspects of AI become a step closer to being reliable and trustworthy.
People |
ORCID iD |
| Raul Santos-Rodriguez (Principal Investigator / Fellow) |
Publications
Fillola E
(2023)
A machine learning emulator for Lagrangian particle dispersion model footprints: a case study using NAME
in Geoscientific Model Development
Bi H
(2022)
An active semi-supervised deep learning model for human activity recognition
in Journal of Ambient Intelligence and Humanized Computing
Wac M
(2025)
Capturing Requirements for a Data Annotation Tool for Intensive Care: Experimental User-Centered Design Study.
in JMIR human factors
Wac M
(2023)
CATS: Cloud-native time-series data annotation tool for intensive care
in SoftwareX
Telmo M. Silva Filho
(2023)
Classifier calibration: a survey on how to assess and improve predicted class probabilities
Silva Filho T
(2023)
Classifier calibration: a survey on how to assess and improve predicted class probabilities
in Machine Learning
Telmo M. Silva Filho
(2023)
Classifier calibration: a survey on how to assess and improve predicted class probabilities
Stawarz K
(2023)
Co-designing opportunities for Human-Centred Machine Learning in supporting Type 1 diabetes decision-making
in International Journal of Human-Computer Studies
Ayobi A
(2021)
Co-Designing Personal Health? Multidisciplinary Benefits and Challenges in Informing Diabetes Self-Care Technologies
in Proceedings of the ACM on Human-Computer Interaction
Kang B
(2021)
Conditional t-SNE: More informative t-SNE embeddings
Kang B
(2021)
Conditional t-SNE: more informative t-SNE embeddings.
in Machine learning
Yamagata T
(2022)
Continuous Adaptation with Online Meta-Learning for Non-Stationary Target Regression Tasks
in Signals
Vosper E
(2023)
Deep Learning for Downscaling Tropical Cyclone Rainfall to Hazard-Relevant Spatial Scales
in Journal of Geophysical Research: Atmospheres
Wac M
(2023)
Design and Evaluation of an Intensive Care Unit Dashboard Built in Response to the COVID-19 Pandemic: Semistructured Interview Study.
in JMIR human factors
Laparra V
(2024)
Estimating Information Theoretic Measures via Multidimensional Gaussianization.
in IEEE transactions on pattern analysis and machine intelligence
Werner E
(2023)
Explainable hierarchical clustering for patient subtyping and risk prediction.
in Experimental biology and medicine (Maywood, N.J.)
Sokol K
(2022)
FAT Forensics: A Python toolbox for algorithmic fairness, accountability and transparency
in Software Impacts
Bi H
(2021)
Human Activity Recognition Based on Dynamic Active Learning.
in IEEE journal of biomedical and health informatics
Poyiadzi, R.
(2021)
Hypothesis Testing for Class-Conditional Label Noise
in arXiv
Yang W.
(2024)
Hypothesis Testing for Class-Conditional Noise Using Local Maximum Likelihood
in Proceedings of the AAAI Conference on Artificial Intelligence
Santos-Rodriguez R
(2021)
Keynote: Training with imperfect and weak labels
Thomas J
(2021)
Multi-lingual agents through multi-headed neural networks
Poyiadzi, R.
(2021)
On the overlooked issue of defining explanation objectives for local-surrogate explainers
in arXiv
Hepburn A.
(2022)
ON THE RELATION BETWEEN STATISTICAL LEARNING AND PERCEPTUAL DISTANCES
in ICLR 2022 - 10th International Conference on Learning Representations
| Description | EPSRC IAA -- Jeff Clark |
| Amount | £15,000 (GBP) |
| Organisation | University of Bristol |
| Sector | Academic/University |
| Country | United Kingdom |
| Start | 03/2024 |
| End | 09/2024 |
| Description | EPSRC IAA offline RL exploration (Taku Yamagata) |
| Amount | £15,000 (GBP) |
| Organisation | University of Bristol |
| Sector | Academic/University |
| Country | United Kingdom |
| Start | 09/2024 |
| End | 02/2025 |
| Description | Global Research and Innovation Programme (GRIP) under the US/UK Statement of Intent (SoI) on Artificial Intelligence R&D |
| Amount | £16,000 (GBP) |
| Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
| Sector | Public |
| Country | United Kingdom |
| Start | 08/2021 |
| End | 03/2022 |
| Description | Facebook internship |
| Organisation | |
| Department | Facebook, UK |
| Country | United Kingdom |
| Sector | Private |
| PI Contribution | This was an internship that then continued as subcontracted work for one of the team members of the Fellowship. This was agreed with one of the Fellowship partners as part of their contribution to the Fellowship (Facebook). However, it took place in a different location (UK vs the original plan, US) topic and time because of COVID. RP, researcher in the fellowship team spent time at Facebook, working with the emotion recognition team to develop new techniques for domain generalisation. |
| Collaborator Contribution | Financial and supervision contribution for RP. |
| Impact | Unpublished technical report: Domain Generalisation for Apparent Emotional Facial Expression Recognition across Age-Groups R Poyiadzi, J Shen, S Petridis, Y Wang, M Pantic arXiv preprint arXiv:2110.09168, 2021 |
| Start Year | 2020 |
| Description | NHS UHBW |
| Organisation | University Hospitals Bristol and Weston NHS Foundation Trust |
| Country | United Kingdom |
| Sector | Hospitals |
| PI Contribution | Research on human-centric approaches for ICU processes and data |
| Collaborator Contribution | The clinical team at the hospitals contributed to the user studies around xAI techniques for hospital environments. |
| Impact | https://arxiv.org/abs/2411.11774 https://journals.sagepub.com/doi/full/10.1177/15353702231214253 |
| Start Year | 2021 |
| Title | CATS: Cloud-native time-series data annotation tool for intensive care |
| Description | The software provides a comprehensive, end-to-end solution to the time-series data annotation and proposes a novel approach for a semi-automated annotation in the cloud. It allows for conducting large-scale, asynchronous data annotation activities across multiple, geographically distributed users. |
| Type Of Technology | Software |
| Year Produced | 2023 |
| Open Source License? | Yes |
| Impact | Collaboration with the Bristol Royal Infirmary mechanical ventilation experts and workshops. |
| Title | IQM-vis |
| Description | IQM-Vis is the first open source toolbox dedicated to analysing human image quality metrics, visualising image distortions and conducting human image perception experiments, all through a simple Python interface. |
| Type Of Technology | Software |
| Year Produced | 2025 |
| Open Source License? | Yes |
| Impact | Use across domains in human centric tasks (vision and audio) and in collaboration with different groups worldwide. |
| URL | https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5093957 |
| Title | What and How of Machine Learning Transparency: Building Bespoke Explainability Tools with Interoperable Algorithmic Components |
| Description | Publication of a paper describing the training materials in The Journal of Open Source Education. |
| Type Of Technology | Software |
| Year Produced | 2022 |
| Impact | Used in AI tutorials, workshops and teaching. |
| URL | https://zenodo.org/record/6395489 |
| Description | IROHMS Future Leaders Academy - Forum on Human-centred Technologies and Society |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Postgraduate students |
| Results and Impact | I participated as a panellists on IROHMS Forum on Human-centred Technologies and Society, Chaired by Professor Stuart Allen and hosted by the Human-centred Technologies and Society Working Group. |
| Year(s) Of Engagement Activity | 2021 |
| URL | https://www.cardiff.ac.uk/artificial-intelligence-robotics-and-human-machine-systems/events/irohms-f... |
| Description | Keynote at 2nd International Conference on Trustworthy AI |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Postgraduate students |
| Results and Impact | I gave a keynote talk, particularly addressed to postgraduate students on AI Trustworthiness and Explainability. |
| Year(s) Of Engagement Activity | 2021 |
| URL | https://events.skoltech.ru/ai-trustworthy |
| Description | Keynote at IEEE Percom (Arduous) |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Keynote talk on the importance of annotations in machine learning within the 5th International Workshop on Annotation of useR Data for UbiquitOUs Systems. This is a different community of researchers (pervasive computing) that I wanted to reach to establish links with traditional machine learning. |
| Year(s) Of Engagement Activity | 2021 |
| URL | https://text2hbm.org/arduous/previous-workshops/arduous-2021/ |
| Description | Talk and panel at the Monash Prato Dialogue: AI Summit 2023 and 2022 |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Other audiences |
| Results and Impact | AI summit with keynote talks and panels focusing on bringing together an international network of experts in the field of AI for Social Good, spanning research disciplines across the humanities and social sciences to the technology, science and engineering fields. |
| Year(s) Of Engagement Activity | 2022,2023 |
| URL | https://www.monash.edu/data-futures-institute/news/events/monash-prato-dialogue-ai-summit-2023 |