A framework for designing Mixed Reality situated learning experiences in cultural heritage sites using AR glasses

Lead Research Organisation: Brunel University
Department Name: Arts and Humanities

Abstract

Walking through a cultural heritage site is an immersive educative experience. The physical space itself and the artefacts that are laid out in it tell many stories of the place's history through audio guides, labels, videos, and interactive touch screens. However, the site remains a static display, which is only revived through the imagination of those who visit it through the things they learn about it. Past research has shown that immersive Mixed Reality (MR) technologies such as mobile Augmented Reality (AR) have the potential to bridge the gap between the past and the present during the visitors' walkthrough, and enhance their learning experience. However, mobile AR relies on handheld devices, poor tracking quality of the user, and the digital content is only superficially super-imposed on the physical world not interacting with the physical objects in a realistic way. AR glass technology, also called direct AR, has the potential to break these barriers and provide significantly increased immersion and a plethora of interaction mechanisms to the visitors, both of which can considerably enhance their learning experience.

The proposed fellowship seeks to explore a design framework for developing engaging, situated learning experiences in cultural heritage places for teenage and adult visitors (in the 12+ age group) with the use of AR glasses. AR glasses are a pair of wearable glasses with the ability to extend the visible physical space using superimposed digital content, including holograms, which is so well positioned that it blends successfully with the physical surroundings. They are an emerging MR technology, which has not yet been adequately researched within the context of cultural heritage education. In collaboration with professionals from heritage education, design, and MR development the project will explore the challenges and opportunities in relation to the use, adoption of and engagement with the AR glasses by visitors of diverse demographics. It will then seek to understand what is lost, gained, transformed, and confused in the visitor's learning experience during the transition from a simple to an augmented walkthrough, and build a design framework based on these insights.

The research outputs will highlight implications for future design approaches for MR experiences within the heritage context. The research will demonstrate in what ways AR glasses increase the immersion as opposed to other MR technologies, and explore their educational challenges and opportunities to create compelling educative MR walkthroughs of a heritage site. Accordingly, the programme of research will investigate the type of MR content that can be built upon the physical and educational capabilities of the device. The collaboration has the potential to promote good practices in using emerging technologies, create value for the cultural organisation that will facilitate its design and the MR developer company that will develop it, and highlight emerging skills in the MR sector.

The proposed fellowship comprises a 3-month period of preliminary research, which will bring heritage professionals, particularly educators, and MR developers together to identify the current landscape in terms of educational experiences using immersive technologies for visitors of the specific age group, discuss how MR can engage visitors through meaningful educational interactions, and create a document of considerations. The MR developers will then use this document to design the content for the MR application. This will be followed by a 3-month period of carrying out user evaluation with the public, iteratively adjusting the MR content based on the received feedback. Finally, a 3-month period of reflection, writing, and dissemination will produce a document introducing the design framework, a Mixed Reality application, two journal articles, one conference paper, a website, and a symposium.

Planned Impact

The non-academic user-groups who will benefit from the research include heritage educators, Mixed Reality developers, interaction designers, and the general public.

The proposed fellowship will benefit heritage educators by enabling them to develop design skills to improve the ways they record, preserve and make available cultural heritage. Through the project's activities heritage educators will gain knowledge in the opportunities that Mixed Reality (MR) presents for situated learning, and develop skills to work with it to design meaningful, fun learning activities. With the predictions that AR glasses will become a prominent immersive technology in the future, such skills will be valuable for creating the future digital museums and bringing in new audiences. The proposed activities will seek to break silos between disciplines, enabling curators, educators, archaeologists, historians, and directors to create a vocabulary for sharing their vision with technologists.

By participating in the co-design process the MR developers will work closely with a multidisciplinary group of people who bring an arts and humanities approach to the design of MR applications, and they will practise iterative design informed by the feedback of the public. AR glass technology is a novel field that is normally driven by engineering rather than design approaches. This experience will have an impact on the industry's perspective in developing future projects for the heritage sector, and other sectors within or outside the creative industries. It will also inspire them to engage with the general public and allow them to develop skills to do so. The symposium at the end of the project, my online posts along the process in related online media, as well as the company's blog posts about the project in their website will extend the impact to the wider MR developer community. In addition, the project is using MR technology and at the same time explores potential designs that appropriate it as a tool for situated learning on a heritage site. The project's outcomes will relate to both the hardware and software development of MR applications. The proposed activities have knowledge exchange at their core and aim to strengthen the links between academic and industrial research and development with direct applications for the benefit of both.

The general public will have access to a new application that provides them with new methods of learning about the history of the site they visit. The final version of the MR application will be available to those who own a Hololens device and wish to experience the past in an immersive way during their visit to Sutton House. The project's methodology and insights will be transferable and applicable to other heritage sites in the UK and globally so the impact on the audience engagement can be scaled considerably. In addition, public engagement is at the heart of the MR application's design methodology to ensure the output is of high quality and a meaningful experience for its end-users, the public. Members of the public, as visitors to the house, will be consulted on the design of the experience, reflect on the design decisions of the project team, and their feedback will be analysed informing further adjustments of the MR content. Through the feedback process they will learn about the project and its research questions and discuss their views and ideas of what they would like to experience. Their participation will help them demystify the working mechanisms behind the design of interactive content while their interaction with the project team on site can inspire the next generation of researchers.

The symposium will help the project develop a network of academic and non-academic audiences that can drive future collaborative research. The project's outcomes will strengthen co-design paradigms for interdisciplinary research and development and exemplify the importance of end-user engagement in the process.

Publications

10 25 50
 
Description The undertaken work developed a design framework for creating compelling and meaningful smart glass learning experience for cultural heritage and evaluated the success of the final product developed using this framework in engaging the public. The proposed framework is based on a methodology that was developed during the project. The methodology introduced dramaturgy as lens through which to approach interdisciplinary collaboration in the specific field of immersive digital heritage and employed methods from theatre for designing the narrative and user experience. It borrowed tools from the practice of doing dramaturgy to approach the narrative design based on affect. It also used these tools to orchestrate and balance the different inputs from the heritage educators, who are experts on the history of the place and curate the learning experience based on the archive, the visitor experience experts, who understand the demographics of the visitors and visitor behaviour, and the smart glass developers, who are experts on the limitations and opportunities of the device. The evaluation of the end product (smart glass experience) tested with members of the public of different demographics proved that the methodology was very successful in conveying the place's history and stories in an engaging way for all ages.

Another finding that derived from the collaboration is the need for a new role in the heritage sector and the extended role of heritage curators as we move towards a future where visitors will be entering a historic place with their own smart glasses. The new role includes a set of design skills and knowledge at the intersection of digital/interaction design, smart glass development (and even immersive technology development) and heritage education. This skillset will allow the person/team to commission and supervise the creation of similar experiences based on the learning objectives of the heritage institution. Their knowledge of the affordances and limitations of digital technologies will allow them to envisage the best approach in order to showcase the required material. At the same time, their ability to work in an inter and trans-disciplinary way will largely support and empower novel ways to engage the visitors, enhance collaborations, and help cultural organisations reach the audiences of the 21st century.
Exploitation Route The outcomes are relevant to interaction designers, heritage professionals and Mixed Reality/Augmented Reality developers who work in the intersection of immersive digital heritage. The design framework which will be published in a journal and in the dedicated website can help designers and cultural organisations approach the creation of similar experiences in a future where smart glasses will be widely used.

The extended roles of curators and educators as well as the necessity of the new role within cultural heritage is part of on going discussion with heritage institutions, and it will be further discussed in the Symposium I organise for the 30th March. In the next few months, and after the Symposium, I will write about this in a blog post and in the dedicated website and disseminate widely. The importance of building such capacity within cultural organisations will be identified as key outcome for further funding. The key skills and knowledge development in relation to this role can also form a Masters degree.

Note: The outcomes of the project, the Symposium on the 30th March, teaser film, and other material will be published in the URL I supplied above in the next few months. The design framework will also be available after the publication of the relevant journal article is out (first article anticipated end of 2020).
Sectors Creative Economy,Digital/Communication/Information Technologies (including Software),Leisure Activities, including Sports, Recreation and Tourism,Culture, Heritage, Museums and Collections

URL https://www.marizadima.space/sutton-house-stories
 
Description There is a strong indication that the developed framework can create compelling immersive cultural heritage learning experiences. The developed experience was received very positively by participants of different ages. This demonstrates that such experiences can attract wide groups of visitors, particularly digital natives, and people in the age group 30-40 which is an age group museums struggle to attract. Attracting more people to museums can consequently impact cultural behaviours, enhance learning and understanding of cultures, and also benefit tourism. The use of this framework by interaction designers working within this field can enrich and empower collaborations with heritage educators. In addition, the formation of new roles at the intersection of heritage education, immersive technology development and interaction design can also boost the creative economy.
First Year Of Impact 2019
Sector Creative Economy,Education,Culture, Heritage, Museums and Collections
Impact Types Cultural,Societal,Economic

 
Title Sutton House Stories MR experience 
Description A Mixed Reality application for Microsoft's Hololens 2 device 
Type Of Technology Webtool/Application 
Year Produced 2019 
Impact The MR application for the Hololens 2 device is used to navigate the immersive experience that was created during the project. The application is a means to an end, its creation process was the main research focus that led to the development of the methodology and the design framework that are the outcomes of this project. Its impact, therefore, is, for the moment, manifested in this. Those who own a Hololens 2 can live the experience in situ at Sutton House. The device's price makes it, for the moment, relatively inaccessible to the wider public so that a larger scale impact can be measured. However, with AR rapidly advancing and devices becoming cheaper this wills cease to be a barrier. The experience was received extremely positively by all participants in the user tests. This is an indicator that the application itself, and similar ones designed using the developed framework, will be successful in engaging visitors in learning about a site's history. 
 
Description Mixed Reality in Cultural Heritage Symposium 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Third sector organisations
Results and Impact I organise this Symposium which will take place on the 30th March and will bring together professionals across cultural heritage, museums and galleries, digital media design and immersive technologies, and serious games, along with a variety of scholars for an afternoon focussed on the development of Mixed Reality applications specifically for Cultural Heritage.
The Symposium has so far 30 registered attendees. I will report back updating this section after the Symposium takes place.
Year(s) Of Engagement Activity 2020
URL https://www.eventbrite.co.uk/myevent?eid=97744290805
 
Description Presentation at the Zip Scene international conference 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact I presented preliminary findings from the project at the international conference on Mixed Reality and Interactive Narrative Zip Scene which took place in Budapest. The conference is the first of its kind to bring together these two areas of research and practice and included prominent scholars and practitioners that do ground breaking research in this field. My presentation was met with a lot of interest, links to my work were included in the web blogs of a few of the delegates and I was also invited to present at a workshop on MR, games and cultural heritage as part of the prestigious DiGRA conference (games studies conference). I find particularly positive the fact that most of the delegates come from the field of theatre/performance and experience design, from which I drew to create the methodology, and they found my approach very useful and innovative.
Year(s) Of Engagement Activity 2019
URL http://zip-scene.mome.hu/