SAIS: Secure AI assistantS

There is an unprecedented integration of AI assistants into everyday life, from the personal AI assistants running in our smart phones and homes, to enterprise AI assistants for increased productivity at the workplace, to health AI assistants. Only in the UK, 7M users interact with AI assistants every day, and 13M on a weekly basis. A crucial issue is how secure AI assistants are, as they make extensive use of AI and learn continually. Also, AI assistants are complex systems with different AI models interacting with each other and with the various stakeholders and the wider ecosystem in which AI assistants are embedded. This ranges from adversarial settings, where malicious actors exploit vulnerabilities that arise from the use of AI models to make AI assistants behave in an insecure way, to accidental ones, where negligent actors introduce security issues or use AIS insecurely. Beyond the technical complexities, users of AI assistants are known to have mental models that are highly incomplete and they do not know how to protect themselves.

SAIS (Secure AI assistantS) is a cross-disciplinary collaboration between the Departments of Informatics, Digital Humanities and The Policy Institute at King's College London, and the Department of Computing at Imperial College London, working with non-academic partners: Microsoft, Humley, Hospify, Mycroft, policy and regulation experts, and the general public, including non-technical users. SAIS will provide an understanding of attacks on AIS considering the whole AIS ecosystem, the AI models used in them, and all the stakeholders involved, particularly focusing on the feasibility and severity of potential attacks on AIS from a strategic threat and risk approach. Based on this understanding, SAIS will propose methods to specify, verify and monitor the security behaviour of AIS based on model- based AI techniques known to provide richer foundations than data-driven ones for explanations on the behaviour of AI-based systems. This will result in a multifaceted approach, including: a) novel specification and verification techniques for AIS, such as methods to verify the machine learning models used by AIS; b) novel methods to dynamically reason about the expected behaviour of AIS to be able to audit and detect any degradation or deviation from that expected behaviour based on normative systems and data provenance; iii) co-created security explanations following a techno-cultural method to increase users' literacy of AIS security in a way that users can comprehend.

SAIS will run a programme of activities with a view to maximising:

*Networking and Community Building on AI Security* by engaging with the other projects funded in this call and the wider academic, industry, government and policy stakeholders and the general public. We aim to engender a sustained and collaborative approach to AI Security to create a community of practice both during and after the funding period for the project. This will include knowledge sharing, best practice and translation activities to enable the community to strengthen and for stakeholder to engage and learn from one another, and work to sustain the community beyond the initial funding. Activities will include community symposium, meetings, and meetups; an online community of practice; the creation of an AI Security Problem Book; and White reports of Use Cases.

*Industry Impact* by working in tandem with four companies: Microsoft, Humley, Hospify, Mycroft, and their real-world Use Cases of personal, enterprise, and health AI assistants. This will be catalysed through participatory and co-creation activities embedded in the research programme, as well as regular individual and plenary partner meetings, and secondments from the research team to the companies. We will also explore opportunities for innovation and commercial exploitation.

*Policy Impact* will be facilitated by the establishment of a Policy Advisory Group with relevant policy experts, think tanks, and civil servants from the Policy Institute's current networks to provide expert advice on potential opportunities and challenges that may arise for policy impact; running a Policy Lab to bring together research, policy and practitioner expertise to assess the evidence base identified in the project and make progress on policy challenges arising from it; and creating a Policy Briefing with summarised Lab's results to be made publicly available and used by the Advisory Group and the Policy Institute to disseminate findings further through their policy networks and as a community resource.

*Public Engagement* to build public awareness, reflexivity, and understanding of AI assistant security by running activities facilitated by KCL's Public Engagement team including: Concept development of a format/concept that can be run to engage publics with; Immersive experience to engage the public with research at the Science Gallery London; Public Workshops right after the immersive experience to bring together the research team with target audiences and enable dialogue between them; and Digital Engagement to maximise reach and target a broad end-user base. A dedicated expert will help the team to plan and evaluate the impact on building public awareness, reflexivity, and understanding about privacy and smart personal assistants.

*Developing SAIS Team's Skills* will be delivered leveraging KCL and Imperial's training centres for staff; secondments in industry; interacting with other projects funded in this call, and the general public, industry, and policy bodies; and ORBIT's one-day foundation course on Responsible Research and Innovation.