Machine Learning for Tomorrow: Efficient, Flexible, Robust and Automated

Lead Research Organisation: University of Cambridge
Department Name: Engineering

Abstract

Artificial intelligence systems have recently led to significant advances in the state-of-the-art in downstream fields including computer vision, speech and natural language processing, and game playing. Although impressive, these advances mask a set of fundamental limitations of the underlying machine learning technology that need to be addressed to unlock gains in a wide variety of applications relevant to industry and society.

These limitations come in four main forms. First current approaches are data-inefficient requiring extremely large and painstakingly curated datasets. Second, they are inflexible solving single tasks that are fixed through time. Third, the current approaches are brittle as performance can degrade catastrophically in the face of noise, missing data or adversarially selected data points. Fourth, the approaches are only semi-automated requiring an expert to design and tune them. These limitations mean that many important application domains are currently out of reach. For example, in medicine we typically have only small and noisy datasets which requires data-efficient and robust machine learning. Providing machine learning as a service requires fully-automated machine learning.

This Prosperity Partnership will develop machine learning that is data-efficient, robust, flexible and automated by leveraging recently developed technology from the University of Cambridge's Machine Learning Group and deep expertise from Microsoft Research Cambridge. This partnership has identified a unique testbed of impactful application domains: health, enterprise tools and games development. This research programme is central to realising Microsoft's vision to empower every developer, organization and individual to innovate and transform the world with AI. Moreover, this area of immediate and wide-ranging national importance, and provides pathways to impact by partnering with one of the world's largest technology companies.

Planned Impact

Artificial intelligence (AI) and machine learning (ML) have the potential to transform many industries and society, but current methods have fundamental limitations that prevent this. The basic research tackled in this open academic program will overcome some of these limitations. The program will therefore have large impact in the UK and globally, through and beyond project partner.

Business benefits.

This partnership will bring global impact to Microsoft. The benefits from unlocking machine learning will be quickly applied and scaled through Microsoft's UK and global customers, partners and billions of users, including through productivity tools (Visual Studio for developers, Microsoft Office and Windows), enterprise services (data, customer relationship management, human resources), hyper-scale cloud computing, hardware, and societal impact (accessibility, global sustainability, humanitarian action, philanthropy).

ML to support tomorrow's computer games: The methods developed in this proposal are particularly useful for enabling and improving AI infused tools via (i) faster personalisation, (ii) integration of more disparate data sources (e.g. jointly from spreadsheets and presentations), and (iii) reliable, robust and fully automated AI.

ML to support tomorrow's healthcare: The project will contribute (i) causal modelling methods that enable data driven decision making for healthcare provider policy, (ii) imaging techniques that can adapt to small numbers of data points, (iii) uncertainty aware, robust methods for medical decision making.

Democratising ML: The methods developed in this proposal will provide: (i) superior autoML systems, (ii) federated machine learning method for distributed ML which could also help support new AI hardware being developed by MS, and (iii) meta-learning methods that are deployable on low-power devices.

UK economic.

Almost every UK organisation uses Microsoft technologies. The fundamental tools developed in the proposal will impact a wide variety of downstream application areas used by consumers, enterprises, SMEs, and public sector organisations. Organisations that are investing in establishing the right approach to AI technology now - specifically, by developing underlying values, ethics, and processes - are shown to be outperforming those that are not by 9%. This research program is necessary to provide the fundamental foundations to ensure that AI and ML technology can be deployed ethically, safely, and at scale in products and services. MSR and the Machine Learning Group have a track record in impactful open source software e.g. Infer.net & GPFlow. The project will train people directly involved in the work. The project's RAs/PhDs/PIs will contribute to University training (undergraduate and graduate ML programmes) and at MSR (e.g. the AI residency and internship programmes). Joint workshops and events will be open to communicate the work and foster the AI community, building on the existing MSR-CUED events.

Societal.

The technological advances in secure, private, interpretable and robust AI will have direct impact on society as these are required to protect important principles and values in our society. ML is currently concentrating power in a privileged few, but this will be mitigated by the automation work package that will improve access to machine learning. This program provides an ambitious joint research initiative to focus on key technical blockers and test them in domains of pertinent application, with the opportunity to accelerate research into industrial deployment via Microsoft, and other organisations through open research.

Moreover, there is an urgent need for postgraduates with training in AI/ML. This research programme will train at least 3 PhDs, 1 RA and 2 SRAs and they will help train undergraduates and postgraduates in machine learning and artificial intelligence.
 
Description We have identified a diverse set of real-world tasks, taken from a number of different application areas, that highlight deficiencies in current machine learning approaches that underpin artificial intelligence. These applications have driven the development of an algorithmic framework that supports Efficient, Flexible, Robust and Automated Machine Learning (EFRA-ML) based on probabilistic meta-learning and transfer learning. The framework has been developed in a series of publications at the top machine learning conferences and the code has been made publicly available and has already been used by hundreds of researchers (it has received over 150 stars from users on GitHub). The framework has been used to advance the state-of-the-art on the following problems.

1) Learning from small numbers of images
Task. The ability to produce object recognition systems that require only a handful of images for training would revolutionise many domains, from the insurer who wants to automatically identify damaged household objects in photos, to citizen-scientists who want to identify animals and plants in photos taken by contributors.
Failure of current approaches. Conventional AI approaches perform very poorly in this setting requiring hundreds of images for each new object that a system needs to recognise. Moreover, in such applications users often gradually add additional images of new and existing items that they would like recognised, which requires the machine learning system to update in an incremental way as new information comes in. Current AI technology catastrophically forgets old examples when used in this setting. Finally, applications often require personalisation to occur on the phone of each user. Current mobile phones cannot support traditional learning (standard training algorithms require too much power and also require gradient computations that are not supported currently) and so resource-efficient machine learning is required.
Our contribution. The EFRA-ML framework has set a new state-of-the-art in few-shot image recognition. We have demonstrated this on benchmarks and have shown how the methods also support learning on resource constrained devices.
Together with Microsoft Research Cambridge we have used these methods to improve object recognition tools for the blind/low-vision community. Our methods are in a pathway to production in Microsoft's Seeing AI app. The app enables a mobile phone to assist a visually impaired user to locate items that they regularly need, but which they sometimes find hard to locate. For example, the user may want to find an item in a shop environment based on photos or videos taken at home of the same type of items they have previously purchased. The machine learning algorithm uses these short videos to train a personalised object recognition system to locate these items in a way which is robust to different environments (home / shop / outdoor etc.). We have helped develop the open ORBIT dataset to promote research in this area. We are now working with Microsoft's Confidential Computing group and Prof. Antti Honkela from the University of Helsinki, an expert in private machine learning, to extend the framework to support private federated machine learning.
In addition, our work in this area directly led to a separate research partnership with Toyota Motor Corporation worth £300,000 to date. The partnership with Toyota is using the software from the Prosperity Partnership and building on the publications to develop systems for self-driving cars that are continuously updating and learning.

2) Personalised education
Task. Artificial intelligence systems have the potential to support the curation of intelligent, personalized learning platforms that ultimately improve the quality of education en masse. Companies such as Duolingo which have hundreds of millions of users, rely on machine learning for supporting personalised language learning. In order to accelerate the development of the basic machine learning technology required to support these types of systems more generally, we have focussed on developing predictive models that can predict a student's score on a new question, based on their answers to previous questions.
Failure of current approaches. Current machine learning approaches again find this general scenario challenging: each year new students, about which we have no or little data, join educational platforms. Current users are constantly providing answers to questions and new questions are added too meaning predictions have to be constantly and quickly revised. This requires machine learning techniques that can handle small data and online or continual learning.
Our contribution. Unfortunately, high quality benchmarks - the sort of which has accelerated and focussed research on object recognition and speech recognition ¬- did not previously exist in this application area. To this end, in partnership with Eedi, a leading educational platform which millions of students interact with daily, members of the Prosperity Partnership team have introduced a new, large-scale, real-world dataset and formulated four data mining tasks on multiple choice questions that mimic real learning scenarios and target various aspects of the above question. We released these tasks at competition setting at the top machine learning conference, Neural Information Processing Systems 2020. The NeurIPS education challenge was entered by nearly 400 teams submitted approximately 4000 submissions, with encouragingly diverse and effective approaches to each of our tasks.
The work on the NeurIPS Education challenge and attention it received led to a partnership with Pinpoint Learning in which the University team are using the tools developed for personalisation in the Partnership and our experience with education datasets to produce systems for automatic personalised assessment and teaching of GCSE maths. These are being deployed into production prototype by Pinpoint Learning and will be tested on thousands of students.

3) Personalised Healthcare and Fairness
Task. Data-efficient learning techniques are essential for many applications in personalised healthcare. In this domain predictive performance is key, but aspects such as robustness and fairness have equal importance. Adaptive clinical trials are a key example - in order to efficiently test new treatments automated decision-making systems are used to determine treatments for the patients entering the trial.
Failure of current approaches. However, in spite of their potential, current approaches to deploying adaptive clinical trials - especially the most advanced - can be shown to be unfair and biased.
Our contribution. We partnered with Dr. Adrian Weller and Dr. Niki Kilbertus, two of Europe's top experts in machine learning and fairness, and Dr. Sofía S. Villar who is an expert in clinical trials. We have published a review paper on 'Multi-disciplinary fairness considerations in machine learning for clinical trials' highlighting fairness problems in current practice. We are now testing new methods for fair adaptive clinical trials.

4) Drug discovery
Task. The drug-development process can be accelerated through computational modelling so that most molecules can be prioritized in silico even without being physically available, and only the molecules most likely to succeed are synthesised and measured.
Failure of current methods. Current machine learning methods require large number of examples for training and such data is typically not available when predicting molecular properties and so the methods fail.
Our contribution. The team's EFRA-ML framework has been successfully transferred by another research team to the world of chemistry where it was demonstrated to be the best performing method to date for predicting molecular properties on a new dataset called FS-MOL that was jointly developed by Microsoft and Novartis. (See here for a blog post about this work.) The dataset and associated small-data machine learning methods are key steps forward in automating the early-stage drug discovery process. This shows that the techniques that the Prosperity Partnership team is developing are fundamental and that they are already impacting diverse fields. We are now working directly with Microsoft Research's AI4Science team to generalise these methods further so that they not only predict molecular properties, but rather can automatically suggest new molecules directly from user-specified desired properties.
Exploitation Route The meta-learning approaches that we have developed are suitable for many commercial applications. Beyond the topics mentioned in the key findings, they are particularly well suited to personalisation and include areas such as recommender systems, personalised image recognition and healthcare prediction problems. They are suitable for implementation on mobile phones allowing apps to adapt to user data.
Sectors Digital/Communication/Information Technologies (including Software),Education,Healthcare
URL https://www.microsoft.com/en-us/research/collaboration/microsoft-research-cambridge-university-machine-learning-initiative/
 
Description Assistive technologies for the visually impaired based on personalised object recognition. The Prosperity Team has worked closely with Microsoft Research (MSR) Cambridge to improve object recognition tools for the blind and low-vision community, which is on a path for exploitation in Microsoft's Seeing AI app. This was possible due to the tight integration between the two research teams. This partnership led to a member of Microsoft's team contributing considerably to our published work [1] and devoting considerable additional researcher time and computational resources to that publication. Our team continues to give advice to the Seeing AI app team. Continual learning methods for self-driving. Our publications on data-efficient machine learning and our talks at workshops and conferences directly led to a research partnership with Toyota Motor Corporation worth £150,000 to date. The partnership is using the software from the Prosperity Partnership and building on the publications to develop systems for self-driving cars that are continuously updating and learning. Personalisation of Predictive Models for Sports. The work on the prosperity partnership directly led to a collaboration with Hawk-Eye Innovations (who develop the Video Assistant Refereeing system) and Sports Interactive Games (who develop the Football Manager game) which is applying the techniques developed in the Prosperity Partnership to develop predictive models of football matches that have been trained on real data, for use in analysis of football matches (e.g. counterfactuals) and in computer games (as AI-trained opponents). Publications and demos from the partnership's work on personalisation and few-shot learning [1-3] were instrumental in leading to a meeting with Hawk-Eye's founder, Paul Hawkins, and a successful pitch. A member of Hawk-Eye's machine learning team is spending 6 months in the group as a visitor working on a proof-of-concept system. Developing Practical Tools for Personalised Education. The work on the NeurIPS Education challenge and attention it received led to a partnership with Pinpoint Learning in which we are using the tools developed for personalisation in the Partnership and our experience with education datasets to produce systems for automatic personalised assessment and teaching of GCSE maths. These are being deployed into production prototype by Pinpoint Learning and will be tested on thousands of students. Microsoft's Partnership with the Cambridge ELLIS Unit. Cambridge has recently become one of the first ELLIS units in the country, thanks in part to large support from Microsoft Research. The success of the Prosperity Partnership was a contributing fact that led to MSR wanting to deepen their existing relationships with our research group and similar ones across the University. The European Laboratory for Learning and Intelligent Systems, ELLIS was started recently in response to the fact that 1. Machine learning is at the heart of a technological and societal artificial intelligence revolution, 2. Europe is not keeping up, 3. The distinction between academic research and industrial labs is vanishing. ELLIS aims to address this by involving the very best European academics while working together closely with basic researchers from industry. The Cambridge ELLIS unit is co-led by the University Co-Investigator, Prof. Miguel Hernandez Lobato. The other UK ELLIS units are based in Edinburgh, UCL and Oxford.
First Year Of Impact 2022
Sector Digital/Communication/Information Technologies (including Software)
Impact Types Societal,Economic
 
Description Real-world Class Incremental Learning
Amount £382,709 (GBP)
Funding ID G120675 
Organisation Toyota Motor Corporation 
Sector Private
Country Japan
Start 02/2023 
End 01/2025
 
Description Toyota Research Grant
Amount £150,000 (GBP)
Organisation Toyota Motor Corporation 
Sector Private
Country Japan
Start 03/2020 
End 03/2022
 
Description Unrestricted Gift from ARM Limited
Amount £50,000 (GBP)
Organisation Arm Limited 
Sector Private
Country United Kingdom
Start 01/2021 
 
Description Unrestricted gift
Amount £25,000 (GBP)
Organisation Arm Limited 
Sector Private
Country United Kingdom
Start 01/2022 
 
Title Fs-mol: A few-shot learning dataset of molecules 
Description Small datasets are ubiquitous in drug discovery as data generation is expensive and can be restricted for ethical reasons (e.g. in vivo experiments). A widely applied technique in early drug discovery to identify novel active molecules against a protein target is modelling quantitative structure-activity relationships (QSAR). It is known to be extremely challenging, as available measurements of compound activities range in the low dozens or hundreds. However, many such related datasets exist, each with a small number of datapoints, opening up the opportunity for few-shot learning after pre-training on a substantially larger corpus of data. At the same time, many few-shot learning methods are currently evaluated in the computer-vision domain. We propose that expansion into a new application, as well as the possibility to use explicitly graph-structured data, will drive exciting progress in few-shot learning. Here, we provide a few-shot learning dataset (FS-Mol) and complementary benchmarking procedure. We define a set of tasks on which few-shot learning methods can be evaluated, with a separate set of tasks for use in pre-training. In addition, we implement and evaluate a number of existing single-task, multi-task, and meta-learning approaches as baselines for the community. We hope that our dataset, support code release, and baselines will encourage future work on this extremely challenging new domain for few-shot learning. 
Type Of Material Database/Collection of data 
Year Produced 2021 
Provided To Others? Yes  
Impact Becoming widely used for work that looks at predicting molecular properties from their chemical composition. 
URL https://openreview.net/forum?id=701FtuyLlAd
 
Title LITE: Memory Efficient Meta-Learning with Large Images 
Description Meta learning approaches to few-shot classification are computationally efficient at test time, requiring just a few optimization steps or single forward pass to learn a new task, but they remain highly memory-intensive to train. This limitation arises because a task's entire support set, which can contain up to 1000 images, must be processed before an optimization step can be taken. Harnessing the performance gains offered by large images thus requires either parallelizing the meta-learner across multiple GPUs, which may not be available, or trade-offs between task and image size when memory constraints apply. We improve on both options by proposing LITE, a general and memory efficient episodic training scheme that enables meta-training on large tasks composed of large images on a single GPU. We achieve this by observing that the gradients for a task can be decomposed into a sum of gradients over the task's training images. This enables us to perform a forward pass on a task's entire training set but realize significant memory savings by back-propagating only a random subset of these images which we show is an unbiased approximation of the full gradient. We use LITE to train meta-learners and demonstrate new state-of-the-art accuracy on the real-world ORBIT benchmark and 3 of the 4 parts of the challenging VTAB+MD benchmark relative to leading meta-learners. LITE also enables meta-learners to be competitive with transfer learning approaches but at a fraction of the test time computational cost, thus serving as a counterpoint to the recent narrative that transfer learning is all you need for few-shot classification. 
Type Of Material Computer model/algorithm 
Year Produced 2021 
Provided To Others? Yes  
Impact It is early days having just released this code. 
URL https://github.com/cambridge-mlg/LITE
 
Title NeurIPS Education Challenge 
Description Digital technologies are becoming increasingly prevalent in education, enabling personalized, high quality education resources to be accessible by students across the world. Importantly, among these resources are diagnostic questions: the answers that the students give to these questions reveal key information about the specific nature of misconceptions that the students may hold. Analyzing the massive quantities of data stemming from students' interactions with these diagnostic questions can help us more accurately understand the students' learning status and thus allow us to automate learning curriculum recommendations. In this competition, you will focus on the students' answer records to these multiple-choice diagnostic questions, with the aim of 1) accurately predicting which answers the students provide; 2) accurately predicting which questions have high quality; and 3) determining a personalized sequence of questions for each student that best predicts the student's answers. These tasks closely mimic the goals of a real-world educational platform and are highly representative of the educational challenges faced today. We provide data from the last two school years (2018-2020) of students' answers to mathematics questions from Eedi, a leading educational platform which millions of students interact with daily around the globe. By participating, you have a chance to make a lasting, real-world impact on the quality of personalized education for millions of students across the world. 
Type Of Material Database/Collection of data 
Year Produced 2020 
Provided To Others? Yes  
Impact This challenge was officially backed by the Neural Information Processing Systems conference, the top machine learning conference. The competition received thousands of submissions and we will be publishing a paper summarising the results and the methods that successful competitors used. 
URL https://competitions.codalab.org/competitions/25449
 
Title ORBIT: A real-world few-shot dataset for teachable object recognition collected from people who are blind or low vision 
Description Object recognition predominately still relies on many high-quality training examples per object category. In contrast, learning new objects from only a few examples could enable many impactful applications from robotics to user personalization. Most few-shot learning research, however, has been driven by benchmark datasets that lack the high variation that these applications will face when deployed in the real-world. To close this gap, we present the ORBIT dataset, grounded in a real-world application of teachable object recognizers for people who are blind/low vision. We provide a full, unfiltered dataset of 4,733 videos of 588 objects recorded by 97 people who are blind/low-vision on their mobile phones, and a benchmark dataset of 3,822 videos of 486 objects collected by 77 collectors. 
Type Of Material Database/Collection of data 
Year Produced 2021 
Provided To Others? Yes  
Impact This database is starting to be widely used by the academic community to test new few-shot object recognition systems. The technology driven by this dataset is being used to improve Microsoft's Seeing AI app https://www.microsoft.com/en-us/ai/seeing-ai 
URL https://doi.org/10.25383/city.14294597
 
Description Collaboration with Hawkeye Innovations 
Organisation Hawk-Eye
Country United Kingdom 
Sector Private 
PI Contribution The work on the prosperity partnership directly led to a collaboration with Hawk-Eye Innovations (who develop the Video Assistant Refereeing system) and Sports Interactive Games (who develop the Football Manager game) which is applying the techniques developed in the Prosperity Partnership to develop predictive models of football matches that have been trained on real data, for use in analysis of football matches (e.g. counterfactuals for broadcasting - what would have happened if the player had done something different) and in computer games (as AI-trained opponents). Publications and demos from the partnership's work on personalisation and few-shot learning were instrumental in leading to a meeting with Hawk-Eye's founder, Paul Hawkins, and a successful pitch. A member of Hawk-Eye's machine learning team is spending 6 months in the group as a visitor working on a proof-of-concept system.
Collaborator Contribution The collaboration is directly using published results and code from the Prosperity Partnership as well as our expertise in personalised machine learning and training of large prediction models.
Impact Outputs will be forthcoming shortly.
Start Year 2022
 
Description Partnership with Microsoft Research 
Organisation Microsoft Research
Department Microsoft Research Cambridge
Country United Kingdom 
Sector Private 
PI Contribution This prosperity partnership is between the University of Cambridge and Microsoft Research.
Collaborator Contribution This prosperity partnership is between the University of Cambridge and Microsoft Research.
Impact All of the outputs from the partnership have involved Microsoft Research, but the most impactful within Microsoft has been our contribution to assistive technologies for the visually impaired based on personalised object recognition. The Prosperity Team has worked closely with Microsoft Research Cambridge to improve object recognition tools for the blind and low-vision community by exploiting our work in Microsoft's Seeing AI app. This was possible due to the tight integration between the two research teams. This partnership led to a member of Microsoft's team contributing considerably to our published work [1] and devoting considerable additional researcher time and computational resources to that publication. Our team continues to give advice to the Seeing AI app team.
Start Year 2020
 
Description School Visit (Amersham) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Schools
Results and Impact I gave a talk to about 400 sixth form students about machine learning
Year(s) Of Engagement Activity 2021