Protecting Against Malicious Use of Image Diffusion Models

The rapid advancement in the capabilities of generative AI has been accompanied by an increase in scope for their misuse. In particular, text-to-image diffusion models such as DALL-E 2 have reached
a point at which high quality images can be generated with ease by a broad spectrum of users, not all of whom may have good intent. The aim of this PhD project is to investigate how to protect against
malicious use of this technology. The goal of a text-to-image diffusion model is to generate realistic images based on a text prompt. These models work by using a forward process which sequentially adds noise to an input image and then learns a mapping to reverse this process. Then, starting with noise, we can use the trained model to denoise the image conditioned on the input text and end up with a output image that appears to come from the same distribution as the input images. Instead of retraining or changing how these models work, we can modify parameters in existing models to change their behaviour to be more desirable. One example use case for this methodology is editing implicit biases found in these models. Implicit biases in the training data for text-to-image diffusion models, can lead to perpetuating social and cultural biases when generating images. As an example, asking a diffusion model for an image of a cow will, with high probability, return an image of a cow in a field even although the environment was never specified. Gender biases are also present in these models which is noticeable when generating pictures of people in certain professions. Since retraining the model to avoid these biases is expensive and time consuming, methods such as TIME look to edit the weights of the model after training in order to reduce the likelihood of a chosen bias occurring. There is still a large amount of analysis to be done into the methodology. Some examples include: How is model performance impacted after editing facts? How can we reduce a broader range of biases in one application of the TIME method? Image manipulation using image diffusion models is another emerging issue. The availability and ease at which these tools can be used allow any image to be edited freely, with little in the way of safeguards to prevent malicious intent. In-painting is the process of taking an existing image and using the model to only generate specific areas of the image. One example of how this can be misused is by editing the background of an image of a person to make it appear as if they were somewhere else. Methodology has been proposed to create protections for images against this process, by adding specific perturbations to the image that cause diffusion models to struggle to generate what is prompted. This project would include an analysis into these techniques and other potential prevention strategies.

In the 2018 Government Office for Science report, 'Computational Modelling: Technological Futures', Greg Clarke, the Secretary of State for Business Energy and Industrial Strategy, wrote "Computational modelling is essential to our future productivity and competitiveness, for businesses of all sizes and across all sectors of the economy". With its focus on computational models, the mathematics that underpin them, and their integration with complex data, the MathSys II CDT will generate diverse impacts beyond academia. This includes impacts on skills, on the economy, on policy and on society.

Impacts on skills.
MathSys II will produce a minimum of 50 PhD graduates to support the growing national demand for advanced mathematical modelling and data analysis skills. The CDT will provide each of them with broad core skills in the MSc, a deep knowledge of their chosen research specialisation in the PhD and a complementary qualification in transferable skills integrated throughout. Graduates will thus acquire the profiles needed to form the next generation of leaders in business, government and academia. They will be supported by an integrated pastoral support framework, including a diverse group of accessible leadership role models. The cohort based environment of the CDT provides a multiplier effect by encouraging cohorts to forge long-lasting professional networks whose value and influence will long outlast the CDT itself. MathSys II will seek to maximise the influence of these networks by providing topical training in Responsible Research and Innovation, by maintaining a robust Equality, Diversity & Inclusion policy, and by integration with Warwick's global network of international partnerships.

Economic impacts.
The research outputs from many MathSys II PhD projects will be of direct economic value to commercial, public sector and charitable external partners. Engagement with CDT partners will facilitate these impacts. This includes co-supervision of PhD and MSc projects, co-creation of Research Study Groups, and a strong commitment to provide placements/internships for CDT students. When commercial innovations or IP are generated, we will work with Warwick Ventures, the commercial arm of the University of Warwick, to commercialise/license IP where appropriate. Economic impact may also come from the creation of new companies by CDT graduates. MathSys II will present entrepreneurship as a viable career option to students. One external partner, Spectra Analytics, was founded by graduates of the preceding Complexity Science CDT, thus providing accessible role models. We will also provide in-house entrepreneurship training via Warwick Ventures and host events by external start-up accelerator Entrepreneur First.

Impacts on policy.
The CDT will influence policy at the national and international level by working with external partners operating in policy. UK examples include Department of Health, Public Health England and DEFRA. International examples include World Health Organisation (WHO) and the European Commission for the Control of Foot-and-mouth Disease (EuFMD). MathSys students will also utilise the recently announced UKRI policy internships scheme.

Impacts on society.
Public engagement will allow CDT students to promote the value of their research to society at large. Aside from social media, suitable local events include DataBeers, Cafe Scientifique, and the Big Bang Fair. MathSys will also promote a socially-oriented ethos of technology for the common good. Concretely, this includes the creation of open-source software, integration of software and data carpentry into our computational and data driven research training and championing open-access to research. We will also contribute to the 'innovation culture and science' strand of Coventry's 2021 City of Culture programme.