Difference-of-Convex Convolutional Neural Networks (DC-CNN)

The goal of computer vision is to impart machines with the ability to see, that is, to understand an image similar to a human. This consists of identifying which object categories are present in an image and where (car, person, trees), their actions (running, driving, sitting), and their relative locations (person inside the car, car above the road). The challenge of computer vision lies in obtaining a powerful discriminative representation of an image that allows us to infer the scene encoded within it. Consider, for instance, the representation of an image as captured by a camera, which consists of color values of each of its pixels. Two images that differ greatly in their color values may still depict the same scene (for example, images of the same location at day and night). At the same time, small changes in the color values may result in a completely different scene where objects have moved considerably. This makes raw color values unsuitable for understanding the scene depicted in the image.

Traditional computer vision approaches have relied on hand-designed representations of an image that are more amenable to scene interpretation. Given such a representation, there exist several principled formulations for learning to interpret scenes, which take advantage of the powerful mathematical programming framework of convex optimization. Convex optimization offers many computational advantages: it scales elegantly with the size of the problem, it provides global optimality, it offers convergence guarantees and it can be parallelised over multiple machines without affecting the accuracy.

Recent years have seen the rise of deep learning, and specifically convolutional neural networks (CNN), which aim to automatically obtain the representation of visual data from a large training set. While an automated approach is highly desirable due to its scalability, it comes with the challenge of solving highly complex non-convex mathematical programs. The aim of our research is to overcome these challenges by finding connections between convex optimization and deep learning. The key observation is that the non-convex programs encountered in deep learning for computer vision have a special structure that is closely related to convexity. Specifically, while the mathematical programs are not convex, they are of a difference-of-convex (DC) form. A DC program can be optimized efficiently by an iterative algorithm, which, at each iteration, solves a convex optimization problem.

Our aim is to exploit the structure of DC-CNNs to design the next generation of algorithms for computer vision. Specifically, we will build customized algorithms that will scale up the dimensionality of the CNN by orders of magnitude while keeping the computational cost low. Our algorithms will retain many of the highly desirable benefits of convex programming (convergence, quality guarantees, elegant scaling, distributed computing) while still allow the automatic estimation of image representations.

The impact of such principled and efficient algorithms is potentially huge. The new CNN architectures that this enables will allow researchers to address significantly more complex visual tasks. For example, a generative network that can provide a set of diverse future frames of a given video sequence, or a intelligent agent that can crawl the web for images and videos and complete the captions in order to bridge the gap between visual data and searchable content. Our research results will be made publicly available via open source software. The project is also likely to have a large academic impact, consolidating the leadership of the UK in machine learning and computer vision.

Recent years have witnessed the deployment of computer vision in many real-world applications. Examples include autonomous navigation from companies such as MobilEye, where it is important to understand the scene captured by the sensors placed in a car, and the Kinect sensor for Microsoft XBox, where the human pose has to be estimated automatially from depth images. Established tech-based companies such as Google, Facebook and Microsoft are releasing new software on an almost daily basis including image search (a standard feature available on almost all search engines these days), camera stablization (e.g. Motion Stills), and 3D reconstruction (e.g. Seene). The UK, and indeed most countries in the world, are witnessing an unprecedented increase in the number of computer vision based start-ups. However, this is just the start, as computer vision now tries to answer significantly more challenging problems such as automatically infer all the high-level components of a scene depicted in a visual samples (images uploaded on Flickr, or videos uploaded on YouTube) in order to bridge the large gap between the amount of visual information and the amount of searchable content on the Internet. As we make more progress in computer vision and related areas of
artificial intelligence, the opportunities for new tech-based business will grow manifolds.

In this context, the proposed research will play a key role in the deployment of the next general of computer vision solutions. One of the limiting factors of the current technology is that the current optimization algorithms used in conjunction with the ubiquitous deep learning framework has several practical and theoretical drawbacks: (i) they are slow, often taking several days to train even on state of the art hardware; (ii) they offer no guarantees of convergence; and (iii) they cannot be easily distributed across multiple computational cores. Our research is geared towards providing a new principled optimization approach that can speed up the training of a deep learning framework significantly, provide conver
gence guarantees, and naturally lend itself to parallel processing. As an upshot, researchers and practitioners will be able to cut down the development time for new applications substantially and even address more challenging tasks such as predicting the several diverse sets of future frames for a given video, or generating high-resolution and realistic visual samples from textual description.

The potential impact of the project is exemplied by the PI's collaboration with Microsoft Research on two projects that rely heavily on deep learning. The first is aimed at learning a deep disciminative or general model for visual data, which can be applied to infer human poses from depth images is highly cluttered environments, or enable gesture based human-computer interaction by inferring the joint locations of a hand. The second is the optimization of code, which would enable programmers to focus completely on the correctness of their code and leave the optimization for speed to a neural network that is capable of adapting the code to a given data set of samples. The PI is also pursuing research on automatically estimating the network architectures with the help of a Google funded PhD student, which would enable even faster deployment of the core machine learning technology to novel applications. The PI will continue to pursue these collaborations to create practical applications of the methodologies developed through the proposal.

Successful completion of this research will have a high international impact. As mentioned above, it will be of interest both to the established tech-based companies as well as the emerging start-ups in the UK and worldwide. In addition, it will also be of great interest to the large academic community focused on deep learning.