The Augmented Telegrapher: Multi-player Mixed Reality in a Museum context

Lead Research Organisation: Falmouth University
Department Name: Games Academy

Abstract

Mixed Reality (MR) is where digital objects can be overlaid onto real world spaces made possible by wearable headset devices such as Microsoft's HoloLens, or where digital objects are overlaid onto a mobile phone's camera image. This technology has the advantage that a user can see the real place they are in at the same time as seeing 'fictional' objects within the same space which can be interacted with. These exciting developments can be mobilised within our Museums and Heritage sites to bring to life those places and their collections, helping to interpret and animate their hidden histories. This project will develop an application of MR for the use of visitors to the Telegraph Museum at Porthcurno in Cornwall, a small museum focused on the history of transatlantic telecommunications and the site of the first telecommunication cable running between Britain and the US. We will work closely with the staff at the Museum and with the Cornwall Museums Partnership to develop an application that will help the museum to attract wider interest from a younger audience who the Museum has had difficulty appealing to.

Our plan is to design a multiplayer game using MR which asks the group to solve a fictional problem that has arisen in the Museum's cable tunnels and which takes place during World War II, when the location was of extreme strategic importance. Employing techniques found in role-playing games, the participant group must find and use virtual objects within the space to solve the problems they encounter. There will be various levels of difficulty in play to help include younger people as well as older, less game literate people. We will use this premise to tell some key stories about the location and bring it to life through interaction using audio, digital objects and characterisation.

Central to our plan is to engage visitors with the Museum's locale and assets as performers (playing a role in a game) thereby engaging them with its rich history. We will encounter technical and design challenges because few applications of MR have yet been used in such contexts and the more superior HoloLens headset is still beyond the reach of many users. Our intention is to record and evaluate our approaches to the challenges we encounter to be shared with other researchers in this and related areas. This knowledge has high value culturally and economically and will help the country to develop the skills and content needed to be at the forefront of MR products both in the fields of games and entertainment, as well as in education and training, healthcare and business. We also hope to deploy the excitement around a MR experience, along with the Museum's place in our technological history, to encourage greater enthusiasm for STEM-based subjects.

Alongside interpreting the museum's assets for new audiences, we will also develop models and approaches that can be used by other small Museums to do the same, and we will employ methods to share our research with the sector. The project is part of a larger endeavour here in Cornwall to develop and support the growth of immersive media design, expertise and infrastructure that will link our efforts to the broader games, film and TV industry.

This bid will help build expertise in the technical and design aspects of developing mixed reality applications. These are needed if we are to create meaningful and vivid interpretational applications that can extend and enrich the experience of visiting a Museum and for visitors to get the most out of the assets and collections that they hold. Even for content creators with expertise in traditional media and VR, the design methodologies for meaningful and compelling MR experiences are lacking and untested at larger scales. Technical and design challenges at the heart of this project require urgent attention if museums are to unlock the considerable power of MR to enthuse and enthral their visitors of all ages.

Planned Impact

Museums and heritage sites can benefit significantly from recent developments in applications for wearable technology to help them interpret for visitors their assets in powerful and appealing ways. Our mixed reality, immersive project is designed with this impact at its heart. With its ability to make history live in the ways that film and games can do, MR applications will create a step change in museums' capability to engage visitors in-situ with their assets, allowing them to overturn their image as static places. Our project is designed to build a location-specific working prototype with significant impact for the Museum. Yet, even with the small scope of this first-stage project, we will also be able to develop the means to generate impact for other Museums and attractions through the creation and evaluation of blueprint methods for exploiting the design potential of this new technology in a location-based and historical context. A core feature of our project proposal is the development of an engaging collective experience for small groups of visitors that is presented as a game. We will draw together and evaluate different technical and design strategies to provide solutions to problems that arise within this new and underdeveloped area.

Our project seeks to develop an application that enlivens and animates place and memory through a 360-degree augmented experience within which the visitor plays a role as an active performer. We are working in a 'live' - rather than academic - context and, as such, learning will be rapid because we will gain immediate feedback from our partner museum. This learning will be the source of impact in the sector as we will share our methods and problems with our various stakeholders and with others working commercially to provide solutions in similar contexts. Local impact for our collaborator, the Telegraph Museum Porthcurno will be demonstrated in the

a) development of new audiences, specifically younger audiences (this critical to the Museum's long-term business success and objectives around increasing engagement with culture beyond the usual niche audience for the collection).
b) exploration and generation of a business model for immersive technologies in smaller, rurally located museums.
c) skills development of staff and volunteers, playing its role in contributing to culture of risk taking and innovation in the museum
d) generating the context within which options for new income streams can contribute to business sustainability
e) raising the profile and visibility for small museums with nationally significant heritage
f) testing new ways to interpret complex, unfamiliar heritage in compelling ways
g) increased learning and enjoyment for museum visitors.

These impact factors are both scalable and valuable across the Creative Industries and Heritage sectors. Some degree of impact is also expected within the game/media industry, advertising and PR, E-Health and Education. The Beneficiaries section of this application explains how less direct impact might be felt in encouraging enthusiasm for STEM subjects in younger visitors/users, fuelled by the collective, interactive problem-solving engagement facilitated using MR.

Impact in the academic sphere will accrue around the development of design strategies for telling collectively experienced gamified stories within a MR environment and in the analysis of the interpretational, semiotic and affective potential of the media in a location-based edutainment context. User testing will generate data that is further likely to impact on subsequent developments, informing the next phase of our immersive research plan in collaboration with the Cornwall Museums Partnership. There will be a regional commercial impact given Falmouth's location in an area of economic development and the project will inform Falmouth University's wider aspirations to develop Cornwall's Immersive/MR design and technical expertise.

Publications

10 25 50
 
Description We explored the affordances of Microsoft's Hololens to blend together virtual and real objects in a narrative context and in a museum context. We evaluated the strengths and weaknesses of the interaction design in mixed reality contexts and significantly discovered that there are problems with multi-user syncing. We therefore devised a system to remedy this and in addition we leveraged tactile interfaces to improve usability (the gesture system was causing user dissatisfaction).
Exploitation Route It will be of use to those looking to develop blended reality and immersive experiences in tourist and museum contexts. The syncing solution will be of use to anyone who wants to use the Hololens in a multi-user context; our software solution is available on Github and may pave the way to more robust syncing solutions.
Sectors Digital/Communication/Information Technologies (including Software),Leisure Activities, including Sports, Recreation and Tourism,Culture, Heritage, Museums and Collections

 
Description They have been used to support the growth of a new business plan for the Telegraph Museum which will now offer an augmented reality experience that will help increase footfall. This constitutes economic and cultural impact. We also disseminated the findings at an academic conference in Helsinki and that data has informed the development of new installations in a range of European museums.
First Year Of Impact 2018
Sector Leisure Activities, including Sports, Recreation and Tourism,Culture, Heritage, Museums and Collections
Impact Types Cultural,Economic

 
Description Follow-on Funding
Amount £56,258 (GBP)
Funding ID AH/S01053X/1 
Organisation Arts & Humanities Research Council (AHRC) 
Sector Public
Country United Kingdom
Start 02/2019 
End 01/2020
 
Title The Augmented Telegrapher 
Description A framework for syncing multiple Microsoft HoloLens devices with an environment and interacting with tangible interfaces within the environment. This framework can be used to develop and deploy multi-player mixed-reality experiences for Microsoft HoloLens, and has been tested in a museum context. 
Type Of Technology Software 
Year Produced 2018 
Open Source License? Yes  
Impact This software has been provisionally deployed as a multiplayer immersive experience in a museum context, and forms the foundation for further commercialization. 
URL https://github.com/Falmouth-Games-Academy/AugmentedTelegrapher