Controllable Image Synthesis

The task of image synthesis is concerned with automatically generating new artificial images and manipulating existing ones. Crucially, these images or synthetic modifications thereof must appear to be realistic to human observers. Over the recent years, machine learning techniques have continued to advance the state-of-the-art for this task of image synthesis--the most successful technique of which is the Generative Adversarial Networks that employ deep neural networks to achieve these ends. The majority of existing research however has had a primary goal of improving the visual quality of such synthetic images, and not of developing a deeper understanding of the generation process. Moreover, as is a common problem with such "black box" deep neural networks, interpreting why or how a particular output was produced is not always so straightforward. Thus, the ability to modify these networks in such a way as to produce a particular desired change to the synthetic images is an important task that has been much more neglected.

Our proposed research will focus on this aspect of controllable image synthesis. We plan to design new methodologies, decompositions, and algorithms to better understand and interpret the inner mechanisms and representations of these networks--leading us to develop ways to better control the image synthesis process for creative ends, and for other downstream tasks.

Applications of our proposed research are highly aligned with EPSRC's research area of "Image and vision computing". In particular, we expect the techniques we develop to have "a high degree of relevance to the creative industries"--for example, such techniques could be utilised to automate film post-production, or common editing tasks performed by photographers.