Study of regularisation methods in machine learning

Lead Research Organisation: University College London
Department Name: Computer Science

Abstract

Over the past decade the availability of powerful computers has opened the doors to the use of machine learning techniques in complex application domains such as those arising in computer vision, speech recognition, computational linguistics, marketing science, and bioinformatics, to mention but a few.A central approach in machine learning which has proved valuable in the above domains consists in computing a function from available data by minimising a regularisation error functional which balances different error/penalty criteria. For example, the regularisation error functional may involves the combination of a data term, measuring the empirical error on the data and a penalty term measuring the function complexity. The goal of this visit, during the period of January--June 2006, is to continue to explore both the theoretical and practical implications of the regularisation approach in machine learning as well as produce a first draft of a book on this topic. Prof. Micchelli shares a strong interest with Dr. Pontil in machine learning and the proposed visit will be the first opportunity for them to work together for an extensive period of time.Prof. Micchelli ranks high among the world leaders in computational mathematics. He has made fundamental contributions to that field, especially to problems concerning approximation, representation and estimation of functions. His work has been influential not only in mainstream mathematics but also in nearby fields, particularly in statistics and computer science. He is in the recent ISI list of 200 mathematicians world-wide who are most highly cited.

Publications

10 25 50
 
Description New properties of certain machine learning algorithms based on the minimisation of regularisation functions.

Among other results, we derive a necessary and sufficient condition for the validity of the Representer Theorem (RT) in machine learning. The RT is a cornerstone result in machine learning which has been applied widely in the field. Previous work provided only a sufficient condition for the
validity of the RT. In this work we prove that this condition is also necessary. This has important consequences on the applicability of the RT to machine learning algorithms. Furthermore, we propose a more general form of the RT which applies to matrix learning problems. Rigour: our
analysis is based on advance tools from linear algebra and it requires some non-trivial technical lemmas which are of independent
Exploitation Route Several researchers working on machine learning have followed up our work.
Sectors Education

 
Description Collaboration with Prof. Charles Micchelli, University at Albany SUNY 
Organisation University at Buffalo
Country United States 
Sector Academic/University 
PI Contribution Prof. Micchelli is an expert in approximation theory and convex analysis
Collaborator Contribution Prof. Micchelli provided key insight into issues pertaining the study of convex learning algorithm, the study of so-called representer theorem and the analysis of certain regularisers which arise in structured sparsity learning problems and in multitask learning.
Impact This is multidisciplinary collaboration across Computer Science and Applied Mathematics Please see the section "Publications" for specific research outputs.