The Design Generator
Lead Research Organisation:
Birkbeck, University of London
Department Name: Film, Media and Cultural Studies
Abstract
The Design Generator project is a collaboration between the Vasari Research Centre for Art and Technology, the Birkbeck Institute for Data Analytics (BIDA), and the V&A Museum. This public engagement initiative will involve extensive experimentation with current Machine Learning algorithms and Neural Network technology to develop an interactive tool for exploring the styles and characteristics of objects in the V&A's collection. The primary planned outcome of the project will be a "Design Generator" tool that allows members of the public to combine key terms from the V&A's collection categories (periods, styles, materials, techniques, etc.) in order to generate images of new, imagined museum objects. For example, the user of the tool might choose to combine terms such as "art nouveau" and "stained glass," or "surrealism" and "tapestry" to produce new, perhaps even improbable objects. The AI tool aims to encourage creative learning and exploration of the defining characteristics of the key periods, styles and materials around which the design museum is organized.
The project actively seeks the help of the public and will culminate in a public-engagement workshop involving a pilot panel made up of 8 members of the V&A public. The hands-on
workshop will introduce participants to the capabilities of AI technologies and their current and potential use within the arts and the museum sector. The workshop will also provide participants with an opportunity to trial the prototype Design Generator application and provide feedback on its development.
The project actively seeks the help of the public and will culminate in a public-engagement workshop involving a pilot panel made up of 8 members of the V&A public. The hands-on
workshop will introduce participants to the capabilities of AI technologies and their current and potential use within the arts and the museum sector. The workshop will also provide participants with an opportunity to trial the prototype Design Generator application and provide feedback on its development.
| Description | The Design Generator was an experiment in using a small museum dataset (the V&A's digitized collection) to train an AI model, in this case a text to image generator. The project used a process of pre-training using an existing publicly available model (ru-DALL-E) and then refining that model using the V&A's much smaller dataset (a process referred to as transfer learning). This experiment proved successful and provides a model for future investigations of generative AI and museum collections. The project also used a method which is considerably more sustainable and less resource intensive than most AI training methods. This public engagement project also provided an opportunity to introduce members of the public to the potential and possible risks of fast developing AI technologies within the contexts of museums and the cultural sector more widely. |
| Exploitation Route | The AI model/tool we produced is publicly available on the code repository GitHub and can be utilized as a starting point for other projects experimenting with the use of generative AI in the context of museum collections. We will be presenting our project and findings at the upcoming New Directions in Museum Analytics at King's College where we hope to generate interest amongst other museum and data researchers. |
| Sectors | Creative Economy Digital/Communication/Information Technologies (including Software) Culture Heritage Museums and Collections |
| URL | http://www7.bbk.ac.uk/vasari/2022/09/24/the-design-generator/ |
| Description | As a public engagement project, the Design Generator reached a significant non-academic audience during the 2022 V&A Digital Design Weekend. We engaged with approximately 300-400 members of the museum public at our permanent stand and approximately 70 participants in our 3 dedicated workshops. Encountering the project was, for most members of the public, an introduction to the potential uses of AI technology as tool for engaging with museum collections. The positive public response to the project during the Digital Design Weekend has encouraged V&A Senior Digital Curator Marc Barto to explore the potential of adapting the Design Generator tool in the creation of a public engagement experience connected to an upcoming V&A exhibition. We are currently in discussions about this possibility. The development of the Design Generator has allowed us to successfully bid for an artist in residence grant from the Experimental Humanities Collaborative Network (funded by the Open Society Foundations). New York-based artist Aarati Akkapeddi will be an artist in residence at the Vasari Research Centre in May and June where they will work with the Design Generator and the Vasari's Capabilities for Collections-funded Deep Learning Workstation in creating an AI artwork exploring the V&A collection. https://thephotographersgallery.org.uk/whats-on/akin-aarati-akkapeddi |
| First Year Of Impact | 2022 |
| Sector | Creative Economy,Culture, Heritage, Museums and Collections |
| Impact Types | Cultural |
| Description | After AI Online Publication |
| Amount | $10,000 (USD) |
| Organisation | Open Society Foundations |
| Sector | Charity/Non Profit |
| Country | United States |
| Start | 12/2024 |
| End | 07/2025 |
| Description | Experimental Humanities Collaborative Network - AI Ethics Teaching Unit (Birkbeck/Al-Quds Bard) |
| Amount | $10,000 (USD) |
| Organisation | Open Society Foundations |
| Sector | Charity/Non Profit |
| Country | United States |
| Start | 01/2023 |
| End | 04/2023 |
| Description | Experimental Humanities Collaborative Network - Artist Residency Grant |
| Amount | $9,000 (USD) |
| Organisation | Open Society Foundations |
| Sector | Charity/Non Profit |
| Country | United States |
| Start | 04/2023 |
| End | 06/2023 |
| Description | Experimental Humanities Collaborative Network - Generative AI and Pedagogy Workshop |
| Amount | $5,000 (USD) |
| Organisation | Open Society Foundations |
| Sector | Charity/Non Profit |
| Country | United States |
| Start | 03/2023 |
| End | 04/2023 |
| Title | Design Generator AI Model/Tool |
| Description | The Design Generator is an AI tool that allows members of the public to combine key terms from the V&A's collection categories (periods, styles, materials, techniques, etc.) in order to generate images of new, imagined museum objects. For example, the user of the tool might choose to combine terms such as "art nouveau" and "stained glass," or "surrealism" and "tapestry" to produce new, perhaps even improbable objects. The AI tool aims to encourage creative learning and exploration of the defining characteristics of the key periods, styles and materials around which the design museum is organized. The publicly available AI model involves a process of "pre-training" and "transfer learning." The model is pre-trained on the open ruDALL-E text to image generator and refined using the V&A collection data through a process of transfer learning. |
| Type Of Material | Improvements to research infrastructure |
| Year Produced | 2022 |
| Provided To Others? | Yes |
| Impact | The model can be used as a starting model for future experiments in working with generative AI and museum collection data. We are currently exploring the potential of future use. |
| URL | https://github.com/ale66/VA-design-generator |
| Title | Design Generator AI Model/Tool |
| Description | The Design Generator is an AI tool that allows members of the public to combine key terms from the V&A's collection categories (periods, styles, materials, techniques, etc.) in order to generate images of new, imagined museum objects. For example, the user of the tool might choose to combine terms such as "art nouveau" and "stained glass," or "surrealism" and "tapestry" to produce new, perhaps even improbable objects. The AI tool aims to encourage creative learning and exploration of the defining characteristics of the key periods, styles and materials around which the design museum is organized. The publicly available AI model involves a process of "pre-training" and "transfer learning." The model is pre-trained on the open ruDALL-E text to image generator and refined using the V&A collection data through a process of transfer learning. |
| Type Of Material | Computer model/algorithm |
| Year Produced | 2022 |
| Provided To Others? | Yes |
| Impact | The model can be used as a starting model for future experiments in working with generative AI and museum collection data. We are currently exploring the potential of future use. |
| URL | https://github.com/ale66/VA-design-generator |
| Description | AI for Cultural Heritage Hub (ArCH) Cambridge |
| Organisation | University of Cambridge |
| Country | United Kingdom |
| Sector | Academic/University |
| PI Contribution | The AI for Cultural Heritage Hub (ArCH) will deploy the convening power of Cambridge's distributed network of collections to create a secure workspace and Community of Practice to empower non-technical users (practitioners and academics) to analyse cultural heritage data securely with AI tools. By encouraging collaboration among curators, researchers, IT professionals, and AI experts, the new hub will prototype adaptive AI solutions to enhance understanding of collections and identify a selection of AI tools to address these problems. Birkbeck and the Vasari Research Centre for Art and Technology are collaborating on one of the ArCH's five case studies, exploring the potential of small scale cultural datasets through the use of techniques of fine tuning and transfer learning. |
| Collaborator Contribution | This project is supported by Accelerate Science and ai@cam, the University's flagship mission on on AI for science, citizens and society. T?his challenge led programme is part of the AI-deas initiative, ai@cam's incubator for challenge-led AI research that tackles major scientific and societal challenge. The project involves a number of contributing partners at Cambridge including Dr. Amelie Roper (Head of Research at Cambridge University Library), Dr. Maya Indira Ganesh (Associate Director of the Leverhulme Centre for the Future of Intelligence), and Tuan Pham (Technical Director of the Cambridge Digital Preservation Programme). |
| Impact | Outputs will be generated during 2025. |
| Start Year | 2024 |
| Description | CapCo Public Engagement Project Collaboration |
| Organisation | Birkbeck, University of London |
| Department | Birkbeck Institute for Data Analyitics |
| Country | United Kingdom |
| Sector | Academic/University |
| PI Contribution | The Vasari Research Centre equipment Upgrade facilitated a successful collaborative funding bid between the Vasari, the Birkbeck Institute for Data Analytics, and the V&A Museum. The AHRC CapCo Public Engagement award was granted for the completion of a project entitled "The Design Generator" - an extensive experimentation with current Machine Learning algorithms to develop an interactive tool for exploring the styles and characteristics of objects in the V&A's collection. As the Principal Investigator, I coordinated the project, organized the data transfer from the V&A, and co-supervised the work of two hired Research Associates (specialists in machine learning). The final outcome of the Design Generator was an AI model, a text to image generator, trained on selected categories of the V&A collection. The RAs worked in the Vasari Research Centre, making use of the equipment purchased through the Vasari Equipment Upgrade grant, including the newly acquired Deep Learning Workstation. The Design Generator model was publicly showcased at the 2023 V&A Digital Design Weekend (23-25 September) through a permanent stand and a series of three public workshops. We continued to be in discussions with the V&A's senior digital producer Marc Barto on using the Design Generator as a public engagement tool for an upcoming V&A exhibition and a special event on generative AI. |
| Collaborator Contribution | Dr. Alessandro Provetti, the director of the Birkbeck Institute for Data Analytics and Reader in Data Analysis at Birkbeck, provided computer science, machine learning, and data analysis expertise on the project. Dr. Provetti co-supervised the hiring and work of the Research Associates. The V&A Research Institute will provided access to V&A image collections and relevant data. V&A Senior Digital Producer Marc Barto facilitated the inclusion of the Design Generator in the V&A Digital Design Weekend. |
| Impact | The collaboration has resulted in a successful funding bid and a completed public engagement project. The project has produced a publicly available AI model and dataset. The reseach team has submitted a journal article to the journal Leonardo detailing the outcomes of the project (under review). The project will be presented in April through a Birkbeck's Artsweek public engagement workshop entitled "Generative AI and the Museum" and as a presentation at the "New Directions in Museum Analytics" conference at King's College London in May. This is a multi-disciplinary collaboration involving computer science (machine learning), museum collections, and digital media studies. |
| Start Year | 2021 |
| Description | CapCo Public Engagement Project Collaboration |
| Organisation | Victoria and Albert Museum |
| Country | United Kingdom |
| Sector | Public |
| PI Contribution | The Vasari Research Centre equipment Upgrade facilitated a successful collaborative funding bid between the Vasari, the Birkbeck Institute for Data Analytics, and the V&A Museum. The AHRC CapCo Public Engagement award was granted for the completion of a project entitled "The Design Generator" - an extensive experimentation with current Machine Learning algorithms to develop an interactive tool for exploring the styles and characteristics of objects in the V&A's collection. As the Principal Investigator, I coordinated the project, organized the data transfer from the V&A, and co-supervised the work of two hired Research Associates (specialists in machine learning). The final outcome of the Design Generator was an AI model, a text to image generator, trained on selected categories of the V&A collection. The RAs worked in the Vasari Research Centre, making use of the equipment purchased through the Vasari Equipment Upgrade grant, including the newly acquired Deep Learning Workstation. The Design Generator model was publicly showcased at the 2023 V&A Digital Design Weekend (23-25 September) through a permanent stand and a series of three public workshops. We continued to be in discussions with the V&A's senior digital producer Marc Barto on using the Design Generator as a public engagement tool for an upcoming V&A exhibition and a special event on generative AI. |
| Collaborator Contribution | Dr. Alessandro Provetti, the director of the Birkbeck Institute for Data Analytics and Reader in Data Analysis at Birkbeck, provided computer science, machine learning, and data analysis expertise on the project. Dr. Provetti co-supervised the hiring and work of the Research Associates. The V&A Research Institute will provided access to V&A image collections and relevant data. V&A Senior Digital Producer Marc Barto facilitated the inclusion of the Design Generator in the V&A Digital Design Weekend. |
| Impact | The collaboration has resulted in a successful funding bid and a completed public engagement project. The project has produced a publicly available AI model and dataset. The reseach team has submitted a journal article to the journal Leonardo detailing the outcomes of the project (under review). The project will be presented in April through a Birkbeck's Artsweek public engagement workshop entitled "Generative AI and the Museum" and as a presentation at the "New Directions in Museum Analytics" conference at King's College London in May. This is a multi-disciplinary collaboration involving computer science (machine learning), museum collections, and digital media studies. |
| Start Year | 2021 |
| Description | Generative AI and the Museum: A Workshop |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Public/other audiences |
| Results and Impact | This workshop was part of Birkbeck's public facing Arts Week showcase. It explored the growth of Generative Artificial Intelligence and its potential impact on museum curating, collections and public engagement. The workshop was an opportunity to introduce members of the public to these emerging technologies, while also showcasing the outcomes of the Design Generator research project. Generative AI can produce unique and detailed images based on written descriptions, opening up new possibilities for engaging with museum collections. During this workshop participants learned about the basics of Generative AI models, while discussing some of the controversies surrounding this technology. Participants had an opportunity to experiment with some of the most powerful AI image tools like DALL-E and Stable Diffusion. And they were also introduced to the AHRC-funded research project The Design Generator, a generative AI model trained on the V&A Museum's collection of design objects. |
| Year(s) Of Engagement Activity | 2023 |
| URL | https://www.bbk.ac.uk/events/remote_event_view?id=36472 |
| Description | V&A Digital Design Weekend 2022 |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Professional Practitioners |
| Results and Impact | We introduced our Design Generator project to the museum public during the three day V&A Digital Design Weekend (23-25 September 2022). The Design Generator, a Birkbeck-based public engagement project, is an AI model trained on the V&A collection that automatically generates images of fictional design objects from text prompts. The AI tool aims to encourage creative learning and exploration of the defining characteristics of the key periods, styles and materials around which the V&A's collection is organized. We operated a permanent interactivev stand showcasing the Design Generator during the three days of the event, engaging with approximately 300 to 400 attendees during the weekend. We also ran three dedicate workshops during the Design Weekend, attended by approximately 70 participants in total. These workshops introduced participants to the wider developments of AI technologies being used in the museum context and allowed them to interact with our Design Generator model. We collected 27 survey responses from workshop participants, with 90% agreeing that they found the workshop very enjoyable and stimulating. We received a number of useful audience comments regarding the potential of generative AI tools for museum collections such as, "Could be a starting point for innovation in design across all sectors...." and "You can (potentially) see all the different categories of the collection in one place rather than walking around the whole area. Provides different experience from connecting with the archives and museum objects, and could reach a different audience from traditional museum goers." |
| Year(s) Of Engagement Activity | 2022 |
| URL | https://www.vam.ac.uk/event/YgbjJRV1V5/design-generator |