Designing Mixed Reality Heritage Performances to Support Decolonisation of Heritage Sites
Lead Research Organisation:
Brunel University London
Department Name: Arts and Humanities
Abstract
The heritage industry has been struggling with the legacy of colonialism in their organisations for decades, however, it has only recently become a priority for curators and programmers to actively challenge colonial power structures embedded in heritage sites. Traditionally museum and heritage site narratives were designed to curate historical events through a particular lens, often fraught with colonial and empirical power structures. At the same time, UK and US based heritage organisations have raised concern over outdated exhibition styles, looking to curatorial innovation to attract more diverse, and younger, audiences. The core research inquiry of this project is to investigate the combination of immersive technologies and immersive performance as tools for decolonisation, that has the potential to attract diverse, 21st century audiences.
The project asks how digital heritage performance, using in particular Mixed Reality technologies, can aid heritage sites in their endeavour to attract new audiences while critically engaging the public with under-represented voices and viewpoints of troubled European and colonial histories. To achieve this goal, the project will design and develop two innovative immersive heritage experiences combining Mixed Reality, in the form of smart glasses, and live performance at heritage sites in the UK and US focusing on under-represented stories from 18th century enslaved and freed slave populations living in London and Deerfield, Massachusetts. The research will further future transatlantic industry innovation by providing heritage workers, and their creative industry partners, with two toolkits to assist the design, implementation, and staff training for the use of MR immersive heritage experiences.
In collaboration with historians, immersive technology designers, digital storytellers, performance makers, cultural heritage researchers, on-site staff, and marginalised communities at each heritage site, the project will explore appropriate design methodologies to create an affective story driven participatory experience that challenges colonial narratives in each site. The aim is to produce a toolkit to enable and support a sustainable design collaboration between heritage organisations and the creative industries in the process of decolonisation. In parallel, reflective design will examine challenges of such collaborations, in terms of digital scholarship, required skills, and technical and logistic aspects. It will then deliver a training toolkit for heritage staff and performers, that can help the heritage industry build digital capacity, create sustainable engagement with hybrid technological experiences as tools for decolonising museum sites, teach them innovative theatrical techniques in working with MR platforms, and explore emerging digital and performance skill sets and roles.
The project asks how digital heritage performance, using in particular Mixed Reality technologies, can aid heritage sites in their endeavour to attract new audiences while critically engaging the public with under-represented voices and viewpoints of troubled European and colonial histories. To achieve this goal, the project will design and develop two innovative immersive heritage experiences combining Mixed Reality, in the form of smart glasses, and live performance at heritage sites in the UK and US focusing on under-represented stories from 18th century enslaved and freed slave populations living in London and Deerfield, Massachusetts. The research will further future transatlantic industry innovation by providing heritage workers, and their creative industry partners, with two toolkits to assist the design, implementation, and staff training for the use of MR immersive heritage experiences.
In collaboration with historians, immersive technology designers, digital storytellers, performance makers, cultural heritage researchers, on-site staff, and marginalised communities at each heritage site, the project will explore appropriate design methodologies to create an affective story driven participatory experience that challenges colonial narratives in each site. The aim is to produce a toolkit to enable and support a sustainable design collaboration between heritage organisations and the creative industries in the process of decolonisation. In parallel, reflective design will examine challenges of such collaborations, in terms of digital scholarship, required skills, and technical and logistic aspects. It will then deliver a training toolkit for heritage staff and performers, that can help the heritage industry build digital capacity, create sustainable engagement with hybrid technological experiences as tools for decolonising museum sites, teach them innovative theatrical techniques in working with MR platforms, and explore emerging digital and performance skill sets and roles.
Publications
Dima Mariza
(2024)
Mixed reality experiences for social impact: a design framework
Bouchard D
(2024)
Authenticity, Risk, and Co-Production: Immersive Digital Media in Decolonial Heritage Practice
in Mimesis
| Title | Jin's Dream |
| Description | Jin's Dream is a world's first Mixed Reality Heritage Performance (MRHP), a participatory promenade theatre fusing live acting, Mixed Reality headsets worn by the audience and taking place in the historic site of Ashley house, Historic Deerfield, MA. Visitors encounter and engage with ideologies, beliefs, and societal frameworks of the era that gave a glimpse of how slavery was embedded and justified within society, while at the same time the performance gave voice to the experiences and cultures of enslaved Africans in Deerfield and those in Jamaica and British Honduras. Audience groups of three to four visitors wear a Microsoft Hololens 2 HMD and participate in the performance as house guests at a specific day in the 1700s guided by live actors in different rooms while in specific moments holographic augmentations enhance live acting. The actors, who are not wearing HMDs, are leading the performance, and interacting with the audience and with the holographic content that the audience sees through a novel IoT sensor network and timeline system. The system is designed to make the interchange seamless and to enable the occurrence of 'magical moments' where the virtual and physical content are in synchrony blurring what is perceived as 'real'. Actors wear an M5Core2 IoT prototyping device, under their costumes, through which they are able to trigger content in the Hololens as well as receive a haptic signal, buzz, that is a cue to continue with acting or make an acknowledgment. The devices are connected to a timeline system that controls what is rendered in the Hololens based on a set of predefined scenes and actions implemented following the narrative. We ran over 30 performances of 'Jin's Dream' lasting for 45 minutes with approximately 80 participants in total. |
| Type Of Art | Performance (Music, Dance, Drama, etc) |
| Year Produced | 2023 |
| Impact | There was a strong experiential learning component for participants who saw the house's history through a decolonial lens and learned about its connections to slavery. Many of the visitors learned more about slavery in the north states whereas those who were already knowledgeable on the house's links to slavery learned more in depth about it. The design of the experience produced several insights that were formulated into three toolkits, a design toolkit, a heritage training toolkit and a theatre toolkit all of which work complimentary as tools and guidelines for similar future co-creation projects. |
| URL | https://www.brunel.ac.uk/research/projects/decolonisation-of-heritage-sites-through-mixed-reality-pe... |
| Title | Sancho's Journey |
| Description | 'Sancho's Journey' is an innovative Mixed Reality Heritage Performance (MRHP) designed for Marble Hill House in London, UK. The experience focuses on the house's financial and material connections to the transatlantic slave trade through objects such as a mahogany staircase, and tea and sugar commodities. Audience groups of three to four visitors wear a Microsoft Hololens 2 HMD and participate in the performance as house guests at a specific day in the 1700s guided by live actors in different rooms while in specific moments holographic augmentations enhance live acting. The actors, who are not wearing HMDs, are leading the performance, and interacting with the audience and with the holographic content that the audience sees through a novel IoT sensor network and timeline system. The system is designed to make the interchange seamless and to enable the occurrence of 'magical moments' where the virtual and physical content are in synchrony blurring what is perceived as 'real'. Actors wear an M5Core2 IoT prototyping device, under their costumes, through which they are able to trigger content in the Hololens as well as receive a haptic signal, buzz, that is a cue to continue with acting or make an acknowledgment. The devices are connected to a timeline system that controls what is rendered in the Hololens based on a set of predefined scenes and actions implemented following the narrative. We ran over 30 performances of 'Sancho's Journey' at Marble Hill House in London lasting for 45 minutes with approximately 80 participants in total. |
| Type Of Art | Performance (Music, Dance, Drama, etc) |
| Year Produced | 2023 |
| Impact | There was a strong experiential learning component for participants who saw the house's history through a decolonial lens and learned about its connections to slavery. The design of the experience produced several insights that were formulated into three toolkits, a design toolkit, a heritage training toolkit and a theatre toolkit all of which work complimentary as tools and guidelines for similar future co-creation projects. |
| URL | https://www.brunel.ac.uk/research/projects/decolonisation-of-heritage-sites-through-mixed-reality-pe... |
| Description | - insights on the application of Mixed Reality and theatre as a tool for decolonisation - insights on the collaboration towards creating Mixed Reality Heritage Performances - A design and development framework has been produced to help heritage organisations and experience designers co-create socially impactful Immersive Reality projects. |
| Exploitation Route | The Design and Development framework provides a backbone guidelines document which can be followed from start to end or pick and choose certain parts ethat are applicable to projects that use Mixed Reality in a heritage setting especially under a social impact directive. |
| Sectors | Creative Economy Culture Heritage Museums and Collections |
| URL | https://mxr4impact.com/ |
| Description | Mixed Reality Impact Activites |
| Amount | £10,000 (GBP) |
| Organisation | University of Essex |
| Country | United Kingdom |
| Start | 09/2022 |
| End | 07/2023 |
| Title | Mixed Reality Heritage Performance Design Framework |
| Description | This design framework offers a set of guidelines for professionals across creative, administrative and production departments in heritage organizations, and theatre companies on how to collaborate with immersive media specialists to design and produce specialized experiences. At the same time it encapsulates the processes required for immersive media specialists to design experiences that empower visitors in heritage sites to discover contextual historic information and serve the set scope and broader mission of the heritage institution. |
| Type Of Material | Improvements to research infrastructure |
| Year Produced | 2024 |
| Provided To Others? | Yes |
| Impact | The toolkit has just been completed so it will take some time to record its impact. isbn 9781908549662 |
| URL | https://mxr4impact.com/ |
| Title | Immersive Mixed Reality Performance technical tools and Artefacts |
| Description | Two sets of technical tools and artefacts were developed for the two exhibitions (a. Ashley House, Historic Deerfield Massachusetts, b. Marble Hill House, London, UK) The developed sets consisted of: 1. Real Time 3D (RT3D) Pipeline 2. Content Production and Integration Pipelines 3. Spatial Interaction Pipelines 4. Networking and Synchronization Systems 5. Real Time Performance Control System Both approaches followed a synchronised playback of the experience on multiple HoloLens devices and used IoT for devices and a real-time performance control system (server) to control the connections and messaging. The communications were developed on OSC. We used Unity as the Spatial Content holder and the orchestration and interactions were driven through IoT and real-time performance control system. Each performance developed a bespoke IoT system for to meet the purposes of the exhibition. Performance a, Deerfield Massachusetts, developed a bespoke real-time 3D spatial interaction artefact for Microsoft HoloLens using Unity. The artefact included a number of 3D, 2D and auditory assets which were optimised for performance and quality of mixed reality. The artefact was optimised and set-up for multi-user synchronised performance. The synchronisation was developed as a Timeline using Abelton software and MaxMSP to act as the real-time performance control system which communicated with the IoT devices and the HoloLenses using OSC signals and messages. Performance b, London UK, developed a bespoke real-time 3D spatial interaction artefact for Microsoft HoloLens using Unity. The artefact included a number of 3D, 2D and auditory assets which were optimised for performance and quality of mixed reality. Some of the assets were created using generative AI tools, notably a portrait of King George II was used to generate a 3D model of the king with AI, which was then rigged and animated using motion capture technology. The artefact was optimised and set-up for multi-user synchronised performance. The synchronisation was developed as a bespoke, general purpose State Machine control system to act as the real-time performance control system which communicated with the IoT devices and the HoloLenses using OSC signals and messages. The control system was developed by the team and is compatible with windows, Mac OS, iOS and Android devices. |
| Type Of Technology | Software |
| Year Produced | 2023 |
| Impact | 1. A technical architecture for developing immersive mixed reality performances. 2. Developing the infrastructure to use Unity only as a content holder, therefore reducing compile and build iterations. This approach allows for all adjustments and fine-tuning to take place without a need to restructure scenes and recompile. It also allows for real-time changes during the performance. 3. An OSC architecture for synchronisation and orchestration of the performance. 4. The technical infrastructure for synchronised performances with minimal servers/control systems. 5. Use of AI algorithms and Generative AI tools in the creative pipeline. |
| Description | Consultancy on the decolonisation process |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | The investigating team worked with a group of young people from English Heritage's Shout Out loud project who consulted the team on the design of the experience. The young people were part of Liverpool's anti-slavery museum outreach programmes. Several activities took place in order to introduce them to the project, familiarise and learn how the cutting edge technologies are used and what are their possibilities, and set up online sessions to follow the design. Example activities were a group visit to London - The Ancestors screening and tour of Marble Hill House, Discussion of design assets with the team, Decolonisation in Heritage Spaces with Dr Leila Kamali, Script feedback session with Holly, evaluation of final performance at Marble Hill and other workshops in between the online design sessions. |
| Year(s) Of Engagement Activity | 2022,2023 |
| Description | Decolonizing Heritage Workshop |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Industry/Business |
| Results and Impact | A one day workshop with Heritage professionals on techniques involving immersive storytelling for decolonizing heritage sites. Hosted at English Heritage, run by project Co-Is, Holly Maples and Dominique Bouchard, and featuring our Mixed Reality Project and other practitioners engaging with live and virtual storytelling to decolonize heritage sites in the heritage industry. |
| Year(s) Of Engagement Activity | 2024 |
| Description | Panelist at the Oxford Brookes University Creative Industries Festival 2023 |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | Mariza Dima presented the project as part of the panel 'Serious Games: Joyful Resistance through Art and Creative Technology' to a room of 20 people. Participants were very interested in the design process for an output that aimed to support the decolonisation, the involvement of cutting edge technologies, as well as the fusion of live acting and immersive tech. There were also questions with regards to impact for the academic sector and specifically those academics who work with the creative industries which was further discussed in the panel 'Beyond academia: Research Impact and Relationship Building in the Creative Industries' in which Dima was also a panellist. Her collaborative aspect of the work in the project drove many of the discussions. |
| Year(s) Of Engagement Activity | 2023 |
| URL | https://www.brookes.ac.uk/research/networks/creative-industries-research-and-innovation/festival-202... |
| Description | Presentation at DRHA 2023 |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Other audiences |
| Results and Impact | Presentation given by the team at the DRHA international conference in Torino, Italy. The team presented the design process of the two performances in the US and UK and the evaluation results. |
| Year(s) Of Engagement Activity | 2023 |
| URL | https://www.consultauniversitariateatro.it/convegno-internazionale-drha-digital-research-in-the-huma... |
| Description | Presentation at the Zip Scene conference 2022 |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | The investigating team presented preliminary findings from the project at the international conference on Mixed Reality and Interactive Narrative Zip Scene which took place in Budapest. The conference is the first of its kind to bring together these two areas of research and practice and included prominent scholars and practitioners that do ground breaking research in this field. Most of the delegates come from the field of theatre/performance and experience design, and the presentation drew a lot of interest from fellow academics. |
| Year(s) Of Engagement Activity | 2022 |
| URL | https://zip-scene.mome.hu/ |
| Description | Webinar Bournemouth Natural Sciences Society |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Dr Dima and Dr Daylamani-Zad gave a webinar presenting the design process and outcomes of the project entitled 'Using 3D imaging to take people back in Time' |
| Year(s) Of Engagement Activity | 2024 |
| Description | Webinar British Computing Society |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Dr Dima and Dr Daylamani-Zad gave an online talk at the British Computing Society - Animation and Games Development special group with the title : Bring your heritage site to life blending mixed reality and immersive theatre |
| Year(s) Of Engagement Activity | 2023 |
| URL | https://www.bcs.org/events-calendar/2023/july/webinar-bring-your-heritage-site-to-life-blending-mixe... |