Mathematics of Adversarial Attacks
Lead Research Organisation:
University of Edinburgh
Department Name: Sch of Mathematics
Abstract
This proposal is built on two observations:
1. Empirical experiments have shown that even the most sophisticated and highly-regarded artificial intelligence (AI) tools can be fooled by carefully constructed examples. For example, given a picture of a dog, we can change the picture in a way that is imperceptible to the human eye but makes the AI system change its mind and categorize the picture as a chicken. Such *adversarial attacks* can be shockingly successful, and they clearly have implications for safety, security and ethics.
2. Although many mathematical scientists are contributing to the exciting and fast-moving body of research in AI and deep learning, the main theoretical focus so far has been on approximation power (can we build systems that satisfy a desired list of properties?) and optimization (what is the best way to fine-tune the network details?).
There is an urgent, unmet need for actionable understanding around adversarial attacks: are they inevitable, are they identifiable, and are they generalizable to other forms of attack?
This motivates the themes of the proposal: Inevitability, Identifiability, and Escalation.
Here are three examples of the types of questions that we will address:
A) Is it inevitable that any AI system will be susceptible to adversarial attack (in which case we should assign resources to identifying attacks rather than attempting to eliminate them)?
B) Typical modern AI hardware is fast but has low accuracy (e.g., each computation may carry only 3 digits); can such imprecision be exploited by new forms of adversarial attack?
C) How secure are AI systems to malicious interventions that, rather than attacking the input data, make covert alterations to the parameters in the system?
We will, for the first time, develop and extend highly relevant ideas from the field of mathematics (numerical analysis and approximation theory) to produce concepts and tools that allow us to appreciate fundamental limitations of AI technology, and identify when these limitations are being exposed; thereby contributing to issues of security, interpretability and accountability.
The proposal will involve a post-doctoral research assistant, who will gain valuable skills in a high-demand area. Also, because issues of trust, privacy and security are central to this project, public engagement activities are built in to the plans. A key route to creating lasting impact is the development of practical case studies that highlight the theory that we develop. This will involve the creation of computer code that uses industry-standard AI platforms and data sets: it is an activity that requires specialist skills in coding and data science, and a qualified software engineer will be employed for this task.
Overall, the ideas emerging from this project will transform our understanding of AI systems by using currently overlooked techniques from computational mathematics. Furthermore, by showing that there are challenges at the heart of AI that can be tackled by computational and applied mathematicians, we plan to transform the scale and quality of research interaction at this important mathematics-computer science interface.
1. Empirical experiments have shown that even the most sophisticated and highly-regarded artificial intelligence (AI) tools can be fooled by carefully constructed examples. For example, given a picture of a dog, we can change the picture in a way that is imperceptible to the human eye but makes the AI system change its mind and categorize the picture as a chicken. Such *adversarial attacks* can be shockingly successful, and they clearly have implications for safety, security and ethics.
2. Although many mathematical scientists are contributing to the exciting and fast-moving body of research in AI and deep learning, the main theoretical focus so far has been on approximation power (can we build systems that satisfy a desired list of properties?) and optimization (what is the best way to fine-tune the network details?).
There is an urgent, unmet need for actionable understanding around adversarial attacks: are they inevitable, are they identifiable, and are they generalizable to other forms of attack?
This motivates the themes of the proposal: Inevitability, Identifiability, and Escalation.
Here are three examples of the types of questions that we will address:
A) Is it inevitable that any AI system will be susceptible to adversarial attack (in which case we should assign resources to identifying attacks rather than attempting to eliminate them)?
B) Typical modern AI hardware is fast but has low accuracy (e.g., each computation may carry only 3 digits); can such imprecision be exploited by new forms of adversarial attack?
C) How secure are AI systems to malicious interventions that, rather than attacking the input data, make covert alterations to the parameters in the system?
We will, for the first time, develop and extend highly relevant ideas from the field of mathematics (numerical analysis and approximation theory) to produce concepts and tools that allow us to appreciate fundamental limitations of AI technology, and identify when these limitations are being exposed; thereby contributing to issues of security, interpretability and accountability.
The proposal will involve a post-doctoral research assistant, who will gain valuable skills in a high-demand area. Also, because issues of trust, privacy and security are central to this project, public engagement activities are built in to the plans. A key route to creating lasting impact is the development of practical case studies that highlight the theory that we develop. This will involve the creation of computer code that uses industry-standard AI platforms and data sets: it is an activity that requires specialist skills in coding and data science, and a qualified software engineer will be employed for this task.
Overall, the ideas emerging from this project will transform our understanding of AI systems by using currently overlooked techniques from computational mathematics. Furthermore, by showing that there are challenges at the heart of AI that can be tackled by computational and applied mathematicians, we plan to transform the scale and quality of research interaction at this important mathematics-computer science interface.
Organisations
People |
ORCID iD |
Desmond Higham (Principal Investigator) |
Publications
Tyukin I
(2021)
The Feasibility and Inevitability of Stealth Attacks
Tyukin I
(2023)
The feasibility and inevitability of stealth attacks
in IMA Journal of Applied Mathematics
Higham D
(2023)
Can We Rely on AI?
Blanchard P
(2021)
Accurately computing the log-sum-exp and softmax functions
in IMA Journal of Numerical Analysis
Beerens L
(2023)
Adversarial ink: componentwise backward error attacks on deep learning
in IMA Journal of Applied Mathematics
Arrigo F
(2022)
Dynamic Katz and related network measures
in Linear Algebra and its Applications
Arrigo F
(2021)
Dynamic Katz and Related Network Measures
Description | Detailed examination of the effect that some of the "shorts cuts" used in large-scale artificial intelligence computing can have upon the accuracy of these tools. Mathematically rigorous analysis of the potential for artificial intelligence system to be fooled buy the equivalent of "optical illusions". |
Exploitation Route | More robust algorithms and more insightful training ands testing of algorithms. |
Sectors | Aerospace Defence and Marine Digital/Communication/Information Technologies (including Software) Healthcare |
URL | https://arxiv.org/abs/2106.13997 |
Description | Engagement with the Alan Turing Institute at the conference https://www.turing.ac.uk/events/interpretability-safety-and-security-ai included debates about interpretability and reliability of Ai systems, where I explained some of the mathematically provable boundaries to AI reliability. |
First Year Of Impact | 2021 |
Sector | Aerospace, Defence and Marine,Energy,Financial Services, and Management Consultancy,Security and Diplomacy |
Impact Types | Economic |
Description | Departmental Seminar |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Regional |
Primary Audience | Professional Practitioners |
Results and Impact | Invited research presentation at University of St Andrews. |
Year(s) Of Engagement Activity | 2007,2023 |
Description | LMS/IMA |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | Invited talk at London Mathematical Society/Society for Industrial and Applied Mathematics joint meeting on 30th September and 1st October. Hosted by the ICMS (Edinburgh), addressing the theme of 'Mathematics in Human Society'. |
Year(s) Of Engagement Activity | 2021 |
URL | https://ima.org.uk/17272/lms-ima-joint-meeting-2021-maths-in-human-society/ |
Description | Plenary Research Talk |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Annual Mathematics Conference of the German Mathematical Society, held in Berlin. |
Year(s) Of Engagement Activity | 2022 |
Description | Public/general lecture |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Public/other audiences |
Results and Impact | Presentation at a "Data" workshop held at the International Centre for Mathematical Sciences, Edinburgh. |
Year(s) Of Engagement Activity | 2022 |
Description | Research talk at Skolkovo |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Invited research talk and round table panel membership at Trustworthy AI 5-7 July Skolkovo, Moscow My attendance was virtual. |
Year(s) Of Engagement Activity | 2012,2021 |
URL | https://events.skoltech.ru/ai-trustworthy#content |
Description | Turing mtg |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Invited research talk at Interpretability, safety and security in AI conference 13-15th December, Alan Turing Institute (virtual attendance) |
Year(s) Of Engagement Activity | 2021 |
URL | https://www.turing.ac.uk/events/interpretability-safety-and-security-ai |
Description | faculty lecture |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Postgraduate students |
Results and Impact | Invited faculty lecture (online) Deep Learning: what could go wrong April 2021 |
Year(s) Of Engagement Activity | 2020,2021 |
URL | https://www.youtube.com/watch?v=yVXtoizLl8U |
Description | research worksop |
Form Of Engagement Activity | A formal working group, expert panel or dialogue |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Other audiences |
Results and Impact | Dagstuhl Seminar on 'Higher-Order Graph Models: From Theoretical Foundations to Machine Learning' (21352) August 29- Sep 1, 2021 |
Year(s) Of Engagement Activity | 2021 |
URL | https://www.dagstuhl.de/en/program/calendar/semhp/?semnr=21352 |