Turing AI Fellowship: Interactive Annotations in AI
Lead Research Organisation:
University of Bristol
Department Name: Engineering Mathematics and Technology
Abstract
With the prevalence of data-hungry deep learning approaches in Artificial Intelligent (AI) as the de facto standard, now more than ever there is a need for labelled data. However, while there have been interesting recent discussions on the definition of readiness levels of data, the same type of scrutiny on annotations is still missing in general: we do not know how or when the annotations were collected or what their inherent biases are. Additionally, there are now forms of annotation beyond standard static sets of labels that call for a formalisation and redefinition of the annotation concept (e.g., rewards in reinforcement learning or directed links in causality).
During this Fellowship we will design and establish the protocols for transparent annotations that empowers the data curator to report on the process, the practitioner to automatically evaluate the value of annotations and the users to provide the most informative and actionable feedback. This Fellowship will address all these through a holistic human-centric research agenda, bridging gaps in fundamental research and public engagement with AI.
The Fellowship aims to lay the foundations for a two-way approach to annotations, where the paradigm is shifted from annotations simply being a resource to them becoming a means for AI systems and humans to interact. The bigger picture is that, with annotations seen as an interface between both entities, we will be in a much better position to guide the relation of trust in between learning systems and users, where users translate their preferences into the learning systems' objective functions. This approach will help produce a much needed transformation in how potentially sensitive aspects of AI become a step closer to being reliable and trustworthy.
During this Fellowship we will design and establish the protocols for transparent annotations that empowers the data curator to report on the process, the practitioner to automatically evaluate the value of annotations and the users to provide the most informative and actionable feedback. This Fellowship will address all these through a holistic human-centric research agenda, bridging gaps in fundamental research and public engagement with AI.
The Fellowship aims to lay the foundations for a two-way approach to annotations, where the paradigm is shifted from annotations simply being a resource to them becoming a means for AI systems and humans to interact. The bigger picture is that, with annotations seen as an interface between both entities, we will be in a much better position to guide the relation of trust in between learning systems and users, where users translate their preferences into the learning systems' objective functions. This approach will help produce a much needed transformation in how potentially sensitive aspects of AI become a step closer to being reliable and trustworthy.
People |
ORCID iD |
| Raul Santos-Rodriguez (Principal Investigator / Fellow) |
Publications
Kang B
(2021)
Conditional t-SNE: more informative t-SNE embeddings.
in Machine learning
Santos-Rodriguez R
(2021)
Keynote: Training with imperfect and weak labels
Poyiadzi, R.
(2021)
Hypothesis Testing for Class-Conditional Label Noise
in arXiv
Bi H
(2021)
Human Activity Recognition Based on Dynamic Active Learning.
in IEEE journal of biomedical and health informatics
McConville R
(2021)
Vesta: A digital health analytics platform for a smart home in a box
in Future Generation Computer Systems
Ayobi A
(2021)
Co-Designing Personal Health? Multidisciplinary Benefits and Challenges in Informing Diabetes Self-Care Technologies
in Proceedings of the ACM on Human-Computer Interaction
Thomas J
(2021)
Multi-lingual agents through multi-headed neural networks
Poyiadzi, R.
(2021)
On the overlooked issue of defining explanation objectives for local-surrogate explainers
in arXiv
Kang B
(2021)
Conditional t-SNE: More informative t-SNE embeddings
Yamagata, T.
(2021)
Reinforcement learning with feedback from multiple humans with diverse skills
in arXiv
Schulz J
(2022)
Uncertainty Quantification of Surrogate Explanations: an Ordinal Consensus Approach
in Proceedings of the Northern Lights Deep Learning Workshop
Wang X
(2022)
Self-play learning strategies for resource assignment in Open-RAN networks
in Computer Networks
Koupai A
(2022)
Self-supervised multimodal fusion transformer for passive activity recognition
in IET Wireless Sensor Systems
Hepburn A.
(2022)
ON THE RELATION BETWEEN STATISTICAL LEARNING AND PERCEPTUAL DISTANCES
in ICLR 2022 - 10th International Conference on Learning Representations
| Description | EPSRC IAA -- Jeff Clark |
| Amount | £15,000 (GBP) |
| Organisation | University of Bristol |
| Sector | Academic/University |
| Country | United Kingdom |
| Start | 03/2024 |
| End | 09/2024 |
| Description | EPSRC IAA offline RL exploration (Taku Yamagata) |
| Amount | £15,000 (GBP) |
| Organisation | University of Bristol |
| Sector | Academic/University |
| Country | United Kingdom |
| Start | 09/2024 |
| End | 02/2025 |
| Description | Global Research and Innovation Programme (GRIP) under the US/UK Statement of Intent (SoI) on Artificial Intelligence R&D |
| Amount | £16,000 (GBP) |
| Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
| Sector | Public |
| Country | United Kingdom |
| Start | 08/2021 |
| End | 03/2022 |
| Description | Facebook internship |
| Organisation | |
| Department | Facebook, UK |
| Country | United Kingdom |
| Sector | Private |
| PI Contribution | This was an internship that then continued as subcontracted work for one of the team members of the Fellowship. This was agreed with one of the Fellowship partners as part of their contribution to the Fellowship (Facebook). However, it took place in a different location (UK vs the original plan, US) topic and time because of COVID. RP, researcher in the fellowship team spent time at Facebook, working with the emotion recognition team to develop new techniques for domain generalisation. |
| Collaborator Contribution | Financial and supervision contribution for RP. |
| Impact | Unpublished technical report: Domain Generalisation for Apparent Emotional Facial Expression Recognition across Age-Groups R Poyiadzi, J Shen, S Petridis, Y Wang, M Pantic arXiv preprint arXiv:2110.09168, 2021 |
| Start Year | 2020 |
| Description | NHS UHBW |
| Organisation | University Hospitals Bristol and Weston NHS Foundation Trust |
| Country | United Kingdom |
| Sector | Hospitals |
| PI Contribution | Research on human-centric approaches for ICU processes and data |
| Collaborator Contribution | The clinical team at the hospitals contributed to the user studies around xAI techniques for hospital environments. |
| Impact | https://arxiv.org/abs/2411.11774 https://journals.sagepub.com/doi/full/10.1177/15353702231214253 |
| Start Year | 2021 |
| Title | CATS: Cloud-native time-series data annotation tool for intensive care |
| Description | The software provides a comprehensive, end-to-end solution to the time-series data annotation and proposes a novel approach for a semi-automated annotation in the cloud. It allows for conducting large-scale, asynchronous data annotation activities across multiple, geographically distributed users. |
| Type Of Technology | Software |
| Year Produced | 2023 |
| Open Source License? | Yes |
| Impact | Collaboration with the Bristol Royal Infirmary mechanical ventilation experts and workshops. |
| Title | IQM-vis |
| Description | IQM-Vis is the first open source toolbox dedicated to analysing human image quality metrics, visualising image distortions and conducting human image perception experiments, all through a simple Python interface. |
| Type Of Technology | Software |
| Year Produced | 2025 |
| Open Source License? | Yes |
| Impact | Use across domains in human centric tasks (vision and audio) and in collaboration with different groups worldwide. |
| URL | https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5093957 |
| Title | What and How of Machine Learning Transparency: Building Bespoke Explainability Tools with Interoperable Algorithmic Components |
| Description | Publication of a paper describing the training materials in The Journal of Open Source Education. |
| Type Of Technology | Software |
| Year Produced | 2022 |
| Impact | Used in AI tutorials, workshops and teaching. |
| URL | https://zenodo.org/record/6395489 |
| Description | IROHMS Future Leaders Academy - Forum on Human-centred Technologies and Society |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Postgraduate students |
| Results and Impact | I participated as a panellists on IROHMS Forum on Human-centred Technologies and Society, Chaired by Professor Stuart Allen and hosted by the Human-centred Technologies and Society Working Group. |
| Year(s) Of Engagement Activity | 2021 |
| URL | https://www.cardiff.ac.uk/artificial-intelligence-robotics-and-human-machine-systems/events/irohms-f... |
| Description | Keynote at 2nd International Conference on Trustworthy AI |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Postgraduate students |
| Results and Impact | I gave a keynote talk, particularly addressed to postgraduate students on AI Trustworthiness and Explainability. |
| Year(s) Of Engagement Activity | 2021 |
| URL | https://events.skoltech.ru/ai-trustworthy |
| Description | Keynote at IEEE Percom (Arduous) |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Keynote talk on the importance of annotations in machine learning within the 5th International Workshop on Annotation of useR Data for UbiquitOUs Systems. This is a different community of researchers (pervasive computing) that I wanted to reach to establish links with traditional machine learning. |
| Year(s) Of Engagement Activity | 2021 |
| URL | https://text2hbm.org/arduous/previous-workshops/arduous-2021/ |
| Description | Talk and panel at the Monash Prato Dialogue: AI Summit 2023 and 2022 |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Other audiences |
| Results and Impact | AI summit with keynote talks and panels focusing on bringing together an international network of experts in the field of AI for Social Good, spanning research disciplines across the humanities and social sciences to the technology, science and engineering fields. |
| Year(s) Of Engagement Activity | 2022,2023 |
| URL | https://www.monash.edu/data-futures-institute/news/events/monash-prato-dialogue-ai-summit-2023 |