A New Generation of Trainable Machines for Multi-Task Learning
Lead Research Organisation:
University College London
Department Name: Computer Science
Abstract
The field of Machine Learning plays an increasingly important role in Computer Science and related disciplines. Over the past decade, the availability of powerful desktop computers has opened the door to synergistic interactions between empirical and theoretical studies of Machine Learning, showing thevalue of the ``learning from example'' paradigm in a wide variety of applications. Much effort has been devoted by Machine Learning researchers to the standard single task learning problem and exciting results have been derived. However, Machine Learning capabilities are still extremely limited when compared to those of humans. The human ability to generalise knowledge learned in one task in order to solve a new task is not available in current Machine Learning systems. Multi-task learning research has not yet received sufficient attention in the field. The standard single task learning approach builds on assumptions that are too restrictive to be easily extended to the novel learning scenarios which are envisaged in this proposal. Although interesting insights on multi-task learning have been provided, at present there is no comprehensive framework for multi-task learning and no cornerstone has yet been placed in the field. Thus, the main purpose of this proposal is to develop this area of Machine Learning research. The proposal focuses on Statistical Machine Learning methods for learning multiple related (classification or regression) tasks and integrating information across them. We shall design formal models of relationships between the tasks and develop (learning algorithms) for learning these relationships from data. We shall also develop the mathematical foundations (generalisation bounds, approximation results, convergence results) for multi-task learning, extending some key theoretical results for single tasklearning. Furthermore, the learning algorithms will be applied to two key applications, namely user preference modelling and multiple microarray gene expression data analysis. A central role in our approach is played by certain graph structures which allow us to model task relationships. This approach is very general and can be adapted to increasingly complex learning scenarios. The computational methods are based on the minimisation of certain penalty functionals via a large number of hyper-parameters associated with the tasks. The proposed research will lead to a new generation of trainable machines for multi-task learning, which will be more powerful and flexible models of learning, closer to human learning than previously developed Machine Learning frameworks.
Organisations
- University College London (Lead Research Organisation)
- INSEAD (Collaboration, Project Partner)
- University at Buffalo (Collaboration)
- University of California, Berkeley (Project Partner)
- University of Cambridge (Project Partner)
- State University of New York (Project Partner)
- Max Planck Inst for Biologogical Cyberne (Project Partner)
- Stanford University (Project Partner)
- University of Southampton (Project Partner)
- RIKEN Center for Brain Science (Project Partner)
- Massachusetts Institute of Technology (Project Partner)
- University of Wisconsin–Madison (Project Partner)
People |
ORCID iD |
Massimiliano Pontil (Principal Investigator) |
Publications
Alexandre Tsybakov
(2009)
Taking advantage of sparsity in multi-task learning
Argyriou A
(2011)
Efficient First Order Methods for Linear Composite Regularizers
Argyriou A
(2008)
Convex multi-task feature learning
in Machine Learning
Argyriou A
(2008)
Machine Learning and Knowledge Discovery in Databases
Argyriou A.
(2009)
When is there a representer theorem? Vector versus matrix regularizers
in Journal of Machine Learning Research
Argyriou A.
(2007)
Multi-task feature learning
in Advances in Neural Information Processing Systems
Argyriou A.
(2006)
Multi-Task Feature Learning
in NIPS 2006: Proceedings of the 19th International Conference on Neural Information Processing Systems
Argyriou A.
(2010)
On spectral learning
in Journal of Machine Learning Research
C Micchelli
(2007)
Representer Theorems for the matrix learning problem.
Caponnetto A
(2008)
Entropy conditions for L r -convergence of empirical processes
in Advances in Computational Mathematics
Description | We developed a novel machine learning framework, which combines information from multiple data sources to improve the problems of learning tasks (e.g. visual object classification, user preference modelling, etc.). Multitask learning (MTL) has been studied since the early 90's. Before the publication of our work, MTL consisted mainly of ad-hoc approaches, lacking a rigorous mathematical foundation. During the course of the project we established a novel framework for MTL based on spectral regularisation and convex optimisation. We provided a detailed analysis of the proposed learning methods and demonstrate numerically that they improve over the state of the art methods. |
Exploitation Route | The techniques developed during the project are now mainstream in machine learning and related disciplines. Some of the papers published are listed as highly cited by ISI Thomson. We continue to work in this area with new publications since the last day of submission |
Sectors | Digital/Communication/Information Technologies (including Software) Healthcare |
Description | Several companies working on Big Data are aware of the importance of multitask and transfer learning; the methodology developed in the project has been used by researchers in some of the leading internet companies such a Google, Yahoo, as well as IBM. |
First Year Of Impact | 2013 |
Sector | Healthcare,Retail |
Impact Types | Economic |
Description | Collaboration with Prof. Charles Micchelli, University at Albany SUNY |
Organisation | University at Buffalo |
Country | United States |
Sector | Academic/University |
PI Contribution | Prof. Micchelli is an expert in approximation theory and convex analysis |
Collaborator Contribution | Prof. Micchelli provided key insight into issues pertaining the study of convex learning algorithm, the study of so-called representer theorem and the analysis of certain regularisers which arise in structured sparsity learning problems and in multitask learning. |
Impact | This is multidisciplinary collaboration across Computer Science and Applied Mathematics Please see the section "Publications" for specific research outputs. |
Description | Collaboration with Prof. Theodoros Evgeniou |
Organisation | INSEAD |
Country | France |
Sector | Private |
PI Contribution | Prof. Evgeniou offers complementaries expertise on practical aspects on machine learning, particularly he is a world leading expert on machine learning applications arising in marketing science and user preference modelling. |
Collaborator Contribution | Prof. Evgeniou discussed with me several multitask learning methodologies as well as provided key insight into their applications to in marketing science and user preference modelling. |
Impact | This is multidisciplinary collaboration across computer science and management science. Please see the section "Publications" for research outputs. |