Creative AI: machine learning as a medium in artistic and curatorial practice

Lead Research Organisation: King's College London
Department Name: Digital Humanities

Abstract

Contemporary art institutions find themselves at the forefront of a wider cultural reflection on critical issues of social and ethical importance. In relation to Artificial Intelligence, however, cultural institutions are in a paradoxical position: on the one side, there is a pressing need to reflect critically and creatively on the ethical and societal impact of AI, and for this AI artworks play a pivotal role. On the other side, cultural institutions often do not have the necessary technical knowledge to support the production of such works, despite the fact that artists such as Trevor Paglen or Adam Harvey have made important contributions by critically using and questioning AI. This project aims to tackle this lack of curatorial knowledge by building tools and guidance for curators to advance institutional media literacy in AI.

Faced with the situation that AI is currently having a transformational impact on our cultures and societies, and given the fact that exhibitions featuring AI artworks are central for delivering critical and ethical inquiries into this technology as well as for advancing a much needed public understanding, our project asks: how can we strengthen curatorial knowledge about AI artworks?

To answer this question and research the missing knowledge of this new artistic medium, our project brings together researchers, artists, cultural practitioners and technologists by connecting the networks of two academic - KCL and NYU - and two contemporary art institutions - Serpentine Galleries and Rhizome/New Museum - situated in London and New York, two centres of both, contemporary art and AI. Each institution has gained its own genuine expertise in the emerging field of Creative AI: In July 2019, Serpentine Galleries have founded a Creative AI lab in collaboration with the Department of Digital Humanities, KCL, with the aim of exploring AI as a tool for artists and gaining a better understanding of the digital skills needed to support critical artistic engagement with this new technology. Around the same time, NYU's Digital Theory lab launched an inquiry into AI Aesthetics, and Rhizome/New Museum started to deliver workshops for artists exploring AI.

Bringing these initiatives into a closer dialogue with one another by linking up their genuine knowledge through collaborative research and a series of workshops, the project will proceed in two steps. Firstly, the project will map existing creative AI tools and explain them targeting curators and artists; and explore how AI technology transforms artistic and with it curatorial practice, to be able to create a systematic introduction into the digital skills and knowledge curators need. Having understood aspects of production of AI artworks, the project will use this knowledge base for a second step, to explore how curators can translate the impact of this new technology into a critical but also creative response. For this, the project systematically studies the social, ethical, and last but not least the aesthetic aspects of contemporary AI.

To explore and verify the above mentioned findings, the project will set up a creative AI experiment that runs parallel to its theoretical explorations. Together with a software engineer, a chatbot prototype will be built, designed intentionally to make aspects of the decision-making process of its AI technology visible through an innovative interface; the interface will reveal decision-making aspects when interacting. The chatbot will be trained on a dataset of interviews which the Serpentine Galleries's Artistic Director Hans Ulrich Obrist has been conducting and recording with artists, architects and thinkers on subjects of art and culture.

Researchers and partners participating in this project share the insight that AI technology profits from and needs more cultural exploration. This project will provide curators with the right skills to support and inquire this technology.

Planned Impact

Overview: This research will directly and immediately benefit anyone working with or about Creative Artificial Intelligence, first and foremost contemporary art galleries as well as more generally cultural institutions interested in involving artworks using Machine Learning; artists working with AI; start-up and computer companies invested in Creative AI; last but not least, our research will engage with and impact the public discourse about AI and ethics by making aspects of AI transparent to a general audience.

Art galleries and cultural institutions: the needs of contemporary art galleries are central to this project and it is here where we will impact most clearly. Some of the project's outcomes - the AI tool map, the webinars, the project website, the reports - are targeted at contemporary art galleries working with AI - to inform their curatorial practice by delivering knowledge relevant for budgeting, managing and commissioning a new artwork involving AI, or by preparing the artwork's exhibition including the information that should be displayed or be covered in a catalogue, be explained in guided tours or introduced at associated events. This information is not just relevant for contemporary art galleries but also for the 1,800 museums registered with the Museum Association, of whom many are interested in using AI in a creative way to playfully process parts of their digitised collection.

Artists: Some of the project's findings inform artists who are working with or would like to work with Machine Learning, for whom the introductory overview of existing tools to produce Creative AI is providing guidance, and who can act on the information of the webinars introducing the calibration of neural networks through interfaces often used by artists, and who want to learn about central issues regarding training data and aesthetic theories and societal topics linked to AI.

Start-up and computer companies: Companies such as Adobe as well as start-ups explore the creative application of Machine Learning and our research into its usage in artworks provides pointers for this industry. Furthermore, the explainability of AI has become a recognised issue, to which our practical research experiment building a chatbot prototype whose interface reveals its decision-making process will contribute.

General public: Given the fact that AI has become an omnipresent technology - from suggesting words when we text on our smartphones to wrongly or rightly identifying criminals for the Metropolitan Police using facial recognition -, the efforts of this project will directly (in its public facing events) and indirectly (through allowing cultural institutions to stage more of them) have an impact on the general public. This is the more the case as the exhibition at the Barbican, London 'AI: More than Human' was highly successful showing that there is a strong interest by the general public to engage with this topic.

Engaging with non-academic players is a profound part of this research project and having impact on the areas above will be actively initiated. We will approach players in the non-academic areas named above, whether this is via inviting them as guest speakers or to come to our events, or to give semi-structured expert interviews (artists, computer industry), or through spreading our findings via the social-media accounts of the four participating partners in this project.
 
Title Dmstfcn, God Mode (ep.1) 
Description This live, sold-out performance at Serpentine Galleries combined live simulation, audio transformation, and score performance. It was also an opportunity to gather audience response data. The Creative AI Lab and Coventry University presents GOD MODE (ep. 1), an interactive audiovisual performance by artist duo dmstfctn exploring the use of simulation in artificial intelligence training. The performance is set within a real-time simulation of a supermarket, a replica of those used to train AI to navigate 3D environments and recognise items on shelves for use in cashier-less supermarkets, like Amazon Fresh. In GOD MODE (ep.1) an AI training in the simulation delivers a fictional monologue that sees it become frustrated with the difficulty of its training, until it finds a bug to exploit in order to complete it and cheat its way out. The simulation is rendered in real time and is navigated by the artists, who also perform facial motion capture and voice modulation of themselves to animate the AI on screen. The audience participates in the training by interacting with the simulation on their phones. A soundtrack performed by artist HERO IMAGE responds to the live, communally orchestrated simulation. GOD MODE (ep.1) explores the increasing use of game engine simulations of 'real world' scenarios to train computer vision systems. By welcoming nuanced interaction with the audience, it aims to reveal the characteristics and limitations of this synthetic reality. GOD MODE (ep.1) is the first episode in dmstfctn's ongoing GOD MODE series and is presented in conjunction with the game GOD MODE: EPOCHS as part of a collaboration between Serpentine's Creative AI lab and Coventry University, supported by the Alan Turing Institute 
Type Of Art Performance (Music, Dance, Drama, etc) 
Year Produced 2022 
Impact This performance operated as a prototypical developmental stage on the way to the development of a full game by dmstfcn comissioned by the Creative AI Lab alongside Center for Postdigital Studies at Coventry University. It was an opportunity to gather audience response and data, as well as exchange knowledge during the Q&A. 
URL https://www.serpentinegalleries.org/whats-on/dmstfctn-god-mode-ep1/
 
Description - We created an online accessible data base that provides an overview and guidance to artistic Machine Learning tools. This has been used by artists and cultural institutions in their production alike available on our website. We also provided access to online sources introducing machine learning in the arts (articles, webinars) which was created by us or curated by us. All findings are available at: https://creative-ai.org/ (Work package 1: mapping creative AI tools)
- Machine learning solicits a shift in the demands placed on artists and curators. Firstly: to build expert teams and complex systems capable of delivering on ambitious projects. Secondly: to themselves develop the technical and theoretical literacy conducive to working effectively within those teams and with emerging technologies. Through a range of studio visits and interviews we found that when it comes to artworks which engage machine learning, collaborations are key - they are numerous, networked, complex. This leads to productive problems of authorship for artists as well as art institutions. Artists coexist and co-create with technical producers and tool-collaborator. (Work package 2: reconfiguring artistic and curatorial practice)
- Early research findings showed that instead of facing a 'black box' artistic work demonstrate that the backend process in which machine learning parameters are adjusted is generally overlooked. We have interviewed artist to understand their production in great detail and found that artistic production of and experimentation with AI is in the exceptional situation inform the debate and production of machine learning in general. We have presented this finding at the International Symposium on Machine Learning and Art 2021, and are in the course of writing this up as a journal article. (Work package 3: contextualising creative AI as a tool of critique).
- We have finished the case studies of AI artworks (based on studio visits with Refik Anadol, Weili Shi, Adam Harvey, Allison Parrish, 0rphan Drift artist collective, Danielle Brathwaite-Shirley). We found that artists work through AI according to two different concepts: one foregrounds the usage of AI processes for visual effects and innovative forms; the other is shining critical light on aspects of AI productions. (Work package 4: AI aesthetics)
- We built a prototype of an interactive chatbot and experimented in collaboration with the software engineer and artist Allison Parrish with the prototype; we have also worked with the designers DVTK in thinking through a public interface. (Work package 5: prototyping AI)
Exploitation Route The outcomes (online resources) provided orientation to other cultural institutions, and cultural practitioners and artists.
The talks we hosted around the topic of the aesthetics of New AI were very well attended by both practitioners and the general audience.
We have build a substantial network and have started to discuss a follow up grant.
Sectors Creative Economy,Education,Culture, Heritage, Museums and Collections

URL https://creative-ai.org/
 
Description 2020/21: Liverpool Biennale is using our data base mapping machine learning tools. Etic Lab LLP is using our repositioning of backend interfaces in their upcoming technology work. Colleagues at NYU are using material from our website https://creative-ai.org/ to inform their upcoming conference. Both online events have had extremely positive feedback with researchers and the general audience confirming that such events were needed and provided much orientation. We are still evaluating both events (due to the pandemic creating a much higher workload this is happening once teaching has finished). The chatbot project has gained the attention of Dr Andrew Starkey from the Computer Science Department, University of Aberdeen working on context analysis. We are currently evaluating the best way and next steps of this collaboration. 2021/22: Our findings - curatorial strategies, conceptual frameworks and technical tool repositories and expert networks - have now been of benefit to a significant number of artists we have consulted with as part of the Lab. This includes the 0rphan Drift artist collective, who have benefited from sustained curatorial support in developing their machine learning project, as well as a dedicated public event. The Serpentine Gallery is currently using our conceptual framework to develop a production prototype with the artist and games designer Danielle Brathwaite-Shirley for their upcoming exhibition. After our online event with Martin Zeilinger, which discussed among others the aspect of copyright and AI, we have received positive feedback from artists who have gainfully attended our events to learn more about how public arts institutions can facilitate AI-artistic research. We have held a workshop with curators and got positive feedback that we helped their cultural institutions to understand the needs of AI artists; participants of our workshops were representatives of the V&A, the Photographer's Gallery, New Inc./New Museum NY, or the Central Academy of Fine Arts, Beijing.
First Year Of Impact 2020
Sector Creative Economy,Digital/Communication/Information Technologies (including Software),Culture, Heritage, Museums and Collections
Impact Types Cultural

 
Description Google AMI Research Award - Tools that Make Meaning
Amount $20,000 (USD)
Organisation Google 
Sector Private
Country United States
Start 03/2021 
End 03/2026
 
Title Creative AI tools and resources database 
Description This database is an ongoing project to aggregate tools and resources for artists, engineers, curators & researchers interested in incorporating machine learning (ML) and other forms of artificial intelligence (AI) into their practice. Resources in the database come from our partners and network; tools cover a broad spectrum of possibilities presented by the current advances in ML like enabling users to generate images from their own data, create interactive artworks, draft texts or recognise objects. 
Type Of Material Database/Collection of data 
Year Produced 2020 
Provided To Others? Yes  
Impact Viewers to this database have a centralized resource where they are able to find all up-to-date ML tools alongside discourse, courses and other resources. In this way, we build a community of people in conversation around this seemly niche subject, which actually touches many fields. We have seen this community grow since the launch of the database; we have received emails from researchers and artists who have found our site and community such as artists Refiek Anadol, Maggie Roberts, Anna Ridler, Holly Grimm, and researchers such as Leo Impett and Memo Akten. 
URL https://creative-ai.org/
 
Description Memorandum CLIMATE JUSTICE AND AI COMPUTING LAB Digital Worlds Institute, University of Florida 
Organisation University of Florida
Country United States 
Sector Academic/University 
PI Contribution In conversation with THE CLIMATE JUSTICE AND AI COMPUTING LAB's principal investigator (Amelia Winger-Bearskin, Digital Worlds Institute, University of Florida) and THE CREATIVE AI LAB'S team (Eva Jäger, Mercedes Bunz), it is clear that we share a similar commitment to including artists and the artistic R&D process in the development of AI technologies. Furthermore, this collaboration will be part of our ongoing research into building public interest technological infrastructure for Web3 with global ecology at the center. Through King's College London's Digital Humanities department's scholarship and the Serpentine's Art Technologies artist network, we plan to co-create artist-led prototypes that can serve as test-beds for future AI technologies.
Collaborator Contribution The AI Climate Justice Lab shared with us insights for prototypes of sustainable technologies during workshops and provide advice to prototypes developed in the lab.
Impact n/a yet.
Start Year 2022
 
Title HUO9000 Semantic Similarity Prototype 
Description Developed by Allison Parrish and the Creative AI Lab, this open-source chatbot prototype allows users to chat with Serpentine's Artistic Director Hans Ulrich Obrist. The dataset (the words spoken by the chatbot) is comprised of a small database of 500 interviews conducted between Hans Ulrich Obrist and various artists between 2015 and the present. (access via http://157.245.129.50:6037 username: huobot password: 9000) 
Type Of Technology Webtool/Application 
Year Produced 2021 
Open Source License? Yes  
Impact Using semantic similarity we were able to take the first steps in developing a new concept of the chatbot-as-archive-interface. As we develop the chatbot, it will serve as a way to navigate through large text databases, like the one we use. 
URL http://157.245.129.50:6037
 
Description 'Creative AI Futures: Theory and practice' (Electronic Visual Arts conference, London, 2022) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact We presented the paper to one of the foremost conferences attended by electronic artists, technical experts and academics.
Year(s) Of Engagement Activity 2022
URL http://www.eva-london.org/eva-london-2022/programme/
 
Description A Cephalopod ? Machine Encounter 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact An in-conversation between Roberts, Alasdair Milne (Creative AI Lab researcher) and Eva Jäger (Associate Curator of Arts Technologies at Serpentine). They discuss 0rphan Drift's latest project, ISCRI, an experiment in interspecies communication between an octopus and an AI, mediated by humans. ISCRI is a collaborative research initiative with technology/machine learning consultancy Etic Lab. Serpentine's Creative AI Lab is a partner on the ISCRI project.

A video work by 0rphan Drift was screened followed by a discussion, the result of sustained research over the previous year. The event was online and we reached over 250 people, across a specialist and general audience.
Year(s) Of Engagement Activity 2021
URL https://www.twitch.tv/videos/1180305964?collection=1C9v8-djsRZtuA
 
Description AI Curatorial Roundtable 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Professional curators, both institutionally-affiliated and freelance, joined the AI Lab for a private roundtable to discuss the challenges of building capacity for curation and display of AI artworks. The session was highly productive for Lab data-gathering purposes, and participants reported that they found the session highly useful too.
Year(s) Of Engagement Activity 2020
 
Description Aesthetics of New AI 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Machine learning technology (a subset study of Artificial Intelligence) has profoundly advanced in the past few years and is becoming a mainstay in artistic practices. Through a discussion on machine learning and its aesthetic capacity, we seek to answer: What new aspects does the technical framework of machine learning bring to art-making? And conversely, what can artworks that use AI point to in AI research and development?
A collaboration with, Serpentine, NYU Digital Theory H-Lab and Rhizome.
Year(s) Of Engagement Activity 2020
URL https://www.serpentinegalleries.org/whats-on/aesthetics-of-new-ai/
 
Description Aesthetics of New AI Panel discussion 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact An international panel discussion with presentations, hosted via zoom, to attend to the questions: "What new aspects does the technical framework of machine learning bring to art-making? And conversely, what can artworks that use AI point to in AI research and development?". The event was well attended with 170 viewers and extensive discussion, with reponses from the panelists. The event was accompanied by a reader compiled of panelists' writing.
Year(s) Of Engagement Activity 2020
URL https://www.serpentinegalleries.org/whats-on/aesthetics-of-new-ai-panel-discussion/
 
Description Aesthetics of New AI: Art making interfaces 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact This second discussion in our 'Aesthetics of New AI' series addresses machine learning technology (a subset study of Artificial Intelligence) through its backend interfaces--what we think of as the internal tooling artists use to develop their work. This event continues our investigation into the changing nature of artistic practices when using new technologies. We will ask: How are interfaces allowing artists to interact with machine learning's meaning-making?
Year(s) Of Engagement Activity 2021
URL https://www.serpentinegalleries.org/whats-on/aesthetics-of-new-ai-interfaces-panel-discussion/
 
Description Between Philosophy of Mind and the Planetary (Transmediale, Berlin, 2023) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Alasdair Milne presented a talk at Transmediale 2023, in addition to engaging with peers during the preparatory exchange sessions: this included an additional presentation, editing and peer-reviewing.
Year(s) Of Engagement Activity 2023
URL https://2023.transmediale.de/en/event/research-workshop-2023-toward-a-minor-tech
 
Description Creative AI - New Inc., New Museum event 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Professional Practitioners
Results and Impact On Wednesday, May 4th we met up with Salome Asega, the Director of New Inc., and her team. New Inc. is a museum-led cultural incubator to exchange artistic and curatorial practices around creative work with AI. We introduced our research findings that were lively discussed and Salome agreed on many points with us - including the dominance of large AI companies as a problem for independent artistic work.
Year(s) Of Engagement Activity 2022
 
Description Creative AI Lab in conversation with Martin Zeilinger (Online Broadcast via Twitch) 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Public/other audiences
Results and Impact This is a live discussion streamed online on Twitch using the Serpentine Gallery's channel. https://www.twitch.tv/videos/1207276107

How do artistic experiments with artificial intelligence impact human-centered notions of creative agency, authorship and ownership?

In his new book, Martin Zeilinger offers a wide-ranging discussion of contemporary digital art practices, and the philosophical and technical considerations of AI. Zeilinger explores posthumanist thought as it relates to emerging issues of intellectual property and the commons. The book invites readers to consider new types of creative practice, rethink the role of the author or singular artist, as well as their originality and unique creative expression.

Join the conversation live via Serpentine's Twitch (@serpentineuk) with Martin Zeilinger, author of Tactical Entanglements (Meson Press, 2021) and Creative AI Lab investigators Mercedes Bunz (King's College London), Eva Jäger (Serpentine) and Daniel Chávez Heras (King's College London).
Year(s) Of Engagement Activity 2021
URL https://www.serpentinegalleries.org/whats-on/tactical-entanglements-creative-ai-lab-in-conversation-...
 
Description Creative AI: Theory and Practice 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Professional Practitioners
Results and Impact Colleagues from disciplines across King's - not just arts and humanities but also engineering and sciences - were invited to discuss what the concept of 'creative AI' means to them and/or to show some specific projects to the audience through which it can be enacted.
Year(s) Of Engagement Activity 2023
URL https://www.kcl.ac.uk/events/creative-ai-theory-and-practice
 
Description Interface Design Workshop Series with DVTK 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact During a series of three workshops focused on interface design, design studio DVTK lead activities that allowed the groups to think through interface design in general and also as it applies to machine learning applications, like the one we are prototyping as part of this grant project, HUO9000. Among other things, we mapped key elements of the user interface, key elements of machine learning to make visible to the user and alternative forms of engagement including spatial interfaces.
Year(s) Of Engagement Activity 2020,2021
 
Description NYU Digital Theory Lab: 'ARTificial Intelligence and Language' 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact The intended purpose was an exchange with experts, practitioners and our US collaborators at NYU and Rhizome/New Museum.

For this workshop, our engineer Allison Parrish presented our experimental project, the chatbot HUO9000 that delves into the nuances of machine learning, especially those involved in replicating human intelligence and interaction through chatbots. The creation of the chatbot can take advantage of an extremely rich data-set unique to our Artistic Director, Hans Ulrich Obrist (HUO): an extensive number of interviews with artists, architects, designers, philosophers, scientists and theorists (over 2,000 hours) that have been the cornerstone to his curatorial practice over the span of his career. We will use this data set to produce a chatbot called HUO9000Beta.

The workshop and our presentation brought us in connection with Dr Andrew Starkey at Aberdeen University, Senior Lecturer in Computer Science, who is now building an interface based on our work and with our input.
Year(s) Of Engagement Activity 2020
 
Description Recurring: Creative AI Lab Theoretical Literature Workshop (Internal) 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Professional Practitioners
Results and Impact This recurring fortnightly meeting creates space for the internal members of the Creative AI Lab team to workshop the most recently emergent literature. The goal of this group is to segment time for intensive close reading, seperate from administrative discussion.

This is extremely valuable time, as it allows the lab to keep up to date with the research that ultimately influences or is included in public-facing activities such as panel discussions.
Year(s) Of Engagement Activity 2020,2021
 
Description Recurring: NYU Digital Theory Lab Reading Group 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact As partners of the Creative AI Lab, the NYU Digital Theory Lab runs frequent reading group workshops, featuring presentations of publications and works-in-progress from international scholars in the field. The reading group operates as a critical research exchange and maintains collaboration across the extended Creative AI Lab network.
Year(s) Of Engagement Activity 2020,2021
 
Description Review of 'Aesthetics of New AI Panel Discussion' for Serpentine Galleries' Art and Ideas platform 
Form Of Engagement Activity Engagement focused website, blog or social media channel
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact Alasdair wrote a critical summary of the panel event 'Aesthetics of New AI' for the Serpentine Galleries' Art and Ideas platform, a public facing editorial platform hosted on the Serpentine's website. Eva and Mercedes provided feedback and support.

The written resource accompanies a video recording of the event to support the institutional history of the galleries' activities. This allows the event to be much more widely accessed than a live-only event, via asynchronous viewing, and the summary allows for the essential elements of the discussion to be gleaned without audiences having to watch the entire 1 hour approx. recording.

A reader compiled by Eva is linked on the page, featuring extended texts by the panel's participants. This allows for audiences to easily access academic writings in a digestible and open-source format.
Year(s) Of Engagement Activity 2020
URL https://www.serpentinegalleries.org/art-and-ideas/aesthetics-of-new-ai/