The Augmented Telegrapher: Multi-player Mixed Reality in a Museum context

Lead Research Organisation: Falmouth University
Department Name: Games Academy

Abstract

Mixed Reality (MR) is where digital objects can be overlaid onto real world spaces made possible by wearable headset devices such as Microsoft's HoloLens, or where digital objects are overlaid onto a mobile phone's camera image. This technology has the advantage that a user can see the real place they are in at the same time as seeing 'fictional' objects within the same space which can be interacted with. These exciting developments can be mobilised within our Museums and Heritage sites to bring to life those places and their collections, helping to interpret and animate their hidden histories. This project will develop an application of MR for the use of visitors to the Telegraph Museum at Porthcurno in Cornwall, a small museum focused on the history of transatlantic telecommunications and the site of the first telecommunication cable running between Britain and the US. We will work closely with the staff at the Museum and with the Cornwall Museums Partnership to develop an application that will help the museum to attract wider interest from a younger audience who the Museum has had difficulty appealing to.

Our plan is to design a multiplayer game using MR which asks the group to solve a fictional problem that has arisen in the Museum's cable tunnels and which takes place during World War II, when the location was of extreme strategic importance. Employing techniques found in role-playing games, the participant group must find and use virtual objects within the space to solve the problems they encounter. There will be various levels of difficulty in play to help include younger people as well as older, less game literate people. We will use this premise to tell some key stories about the location and bring it to life through interaction using audio, digital objects and characterisation.

Central to our plan is to engage visitors with the Museum's locale and assets as performers (playing a role in a game) thereby engaging them with its rich history. We will encounter technical and design challenges because few applications of MR have yet been used in such contexts and the more superior HoloLens headset is still beyond the reach of many users. Our intention is to record and evaluate our approaches to the challenges we encounter to be shared with other researchers in this and related areas. This knowledge has high value culturally and economically and will help the country to develop the skills and content needed to be at the forefront of MR products both in the fields of games and entertainment, as well as in education and training, healthcare and business. We also hope to deploy the excitement around a MR experience, along with the Museum's place in our technological history, to encourage greater enthusiasm for STEM-based subjects.

Alongside interpreting the museum's assets for new audiences, we will also develop models and approaches that can be used by other small Museums to do the same, and we will employ methods to share our research with the sector. The project is part of a larger endeavour here in Cornwall to develop and support the growth of immersive media design, expertise and infrastructure that will link our efforts to the broader games, film and TV industry.

This bid will help build expertise in the technical and design aspects of developing mixed reality applications. These are needed if we are to create meaningful and vivid interpretational applications that can extend and enrich the experience of visiting a Museum and for visitors to get the most out of the assets and collections that they hold. Even for content creators with expertise in traditional media and VR, the design methodologies for meaningful and compelling MR experiences are lacking and untested at larger scales. Technical and design challenges at the heart of this project require urgent attention if museums are to unlock the considerable power of MR to enthuse and enthral their visitors of all ages.

Planned Impact

Museums and heritage sites can benefit significantly from recent developments in applications for wearable technology to help them interpret for visitors their assets in powerful and appealing ways. Our mixed reality, immersive project is designed with this impact at its heart. With its ability to make history live in the ways that film and games can do, MR applications will create a step change in museums' capability to engage visitors in-situ with their assets, allowing them to overturn their image as static places. Our project is designed to build a location-specific working prototype with significant impact for the Museum. Yet, even with the small scope of this first-stage project, we will also be able to develop the means to generate impact for other Museums and attractions through the creation and evaluation of blueprint methods for exploiting the design potential of this new technology in a location-based and historical context. A core feature of our project proposal is the development of an engaging collective experience for small groups of visitors that is presented as a game. We will draw together and evaluate different technical and design strategies to provide solutions to problems that arise within this new and underdeveloped area.

Our project seeks to develop an application that enlivens and animates place and memory through a 360-degree augmented experience within which the visitor plays a role as an active performer. We are working in a 'live' - rather than academic - context and, as such, learning will be rapid because we will gain immediate feedback from our partner museum. This learning will be the source of impact in the sector as we will share our methods and problems with our various stakeholders and with others working commercially to provide solutions in similar contexts. Local impact for our collaborator, the Telegraph Museum Porthcurno will be demonstrated in the

a) development of new audiences, specifically younger audiences (this critical to the Museum's long-term business success and objectives around increasing engagement with culture beyond the usual niche audience for the collection).
b) exploration and generation of a business model for immersive technologies in smaller, rurally located museums.
c) skills development of staff and volunteers, playing its role in contributing to culture of risk taking and innovation in the museum
d) generating the context within which options for new income streams can contribute to business sustainability
e) raising the profile and visibility for small museums with nationally significant heritage
f) testing new ways to interpret complex, unfamiliar heritage in compelling ways
g) increased learning and enjoyment for museum visitors.

These impact factors are both scalable and valuable across the Creative Industries and Heritage sectors. Some degree of impact is also expected within the game/media industry, advertising and PR, E-Health and Education. The Beneficiaries section of this application explains how less direct impact might be felt in encouraging enthusiasm for STEM subjects in younger visitors/users, fuelled by the collective, interactive problem-solving engagement facilitated using MR.

Impact in the academic sphere will accrue around the development of design strategies for telling collectively experienced gamified stories within a MR environment and in the analysis of the interpretational, semiotic and affective potential of the media in a location-based edutainment context. User testing will generate data that is further likely to impact on subsequent developments, informing the next phase of our immersive research plan in collaboration with the Cornwall Museums Partnership. There will be a regional commercial impact given Falmouth's location in an area of economic development and the project will inform Falmouth University's wider aspirations to develop Cornwall's Immersive/MR design and technical expertise.

Publications

10 25 50
 
Description We explored the affordances of Microsoft's Hololens to blend together virtual and real objects in a narrative context and in a museum context. We evaluated the strengths and weaknesses of the interaction design in mixed reality contexts and significantly discovered that there are problems with multi-user syncing. We therefore devised a system to remedy this and in addition we leveraged tactile interfaces to improve usability (the gesture system was causing user dissatisfaction). We are in the final stages of user-testing the Augmented Reality experience on location at Porthcurno Telegraph Museum and it will available to use in April 2020.
Exploitation Route It will be of use to those looking to develop blended reality and immersive experiences in tourist and museum contexts. The syncing solution will be of use to anyone who wants to use the Hololens in a multi-user context; our software solution is available on Github and may pave the way to more robust syncing solutions.
Sectors Digital/Communication/Information Technologies (including Software),Leisure Activities, including Sports, Recreation and Tourism,Culture, Heritage, Museums and Collections

 
Description They have been used to support the growth of a new business plan for the Telegraph Museum which will now offer an augmented reality experience that will help increase footfall. This constitutes economic and cultural impact. We also disseminated the findings at an academic conference in Helsinki and that data has informed the development of new installations in a range of European museums. In addition, we have submitted a paper on our working methodology and the implications for heritage to a special edition of Computing and Heritage and presented a paper on the work's implication for our understanding of historical time and space at a conference in Oxford.We have shared our knowledge with the regional heritage providers and our findings have been used within our an additional project to extend our collaboration with Cornish heritage providers.
First Year Of Impact 2019
Sector Leisure Activities, including Sports, Recreation and Tourism,Culture, Heritage, Museums and Collections
Impact Types Cultural,Economic

 
Description LEP focus on supporting Immersive to support Tourist/heritage sector
Geographic Reach Local/Municipal/Regional 
Policy Influence Type Influenced training of practitioners or researchers
Impact The project alerted the Cornwall Council and the Cornwall and IoS Local Enterprise Partnership to the cultural and economic value of new immersive technologies. The LEP & CC have now included that in the regional Industrial Strategy and we have become partners on a further successful bid submitted to the Coastal Community Fund that develops the work of the Augmented Telegrapher project across five museums (stated April 2019). The Augmented Telegrapher generated and developed skills both in terms of design and technology for the collaborators, helping to equip regional museums, both large and small, to help take advantage of emergent opportunities within the digital economy in a rural area. CMP gained more knowledge about how to support the Museums in the partnership to identify markets, develop new business models and to develop digital skills. In addition, the project has created an eco-system and workflow that has enabled greater and more successful co-design across disciplines and between the heritage sector and design research. This, in turn, is designed to increase footfall for the participating museums creating economic impact for those museums and surrounding businesses.
URL https://www.businesscornwall.co.uk/news-by-industry/public-sector-news-categories/2019/03/700k-for-i...
 
Description Follow-on Funding
Amount £56,258 (GBP)
Funding ID AH/S01053X/1 
Organisation Arts & Humanities Research Council (AHRC) 
Sector Public
Country United Kingdom
Start 02/2019 
End 01/2020
 
Description Porthcurno Telegraph Museum 
Organisation Porthcurno Telegraph Museum
Country United Kingdom 
Sector Charity/Non Profit 
PI Contribution We worked with the Museum team to craft an Escape Room experience that helps to tell the story of the activities undertaken at the Telegraph Station during WW2 with the aim of creating an experience appealing to harder to reach audiences and providing a model that can help drive additional footfall in the down season.
Collaborator Contribution The museum staff bought historical and archival knowledge vital to the task of making a relevant augmented experiences for visitors. We learned about museums' priorities and their audiences. The collaboration enabled by this funding has formed the basis of an ongoing relationship between CMP, the Museum and the Falmouth research team. We collectively applied with the Cornwall and IOS LEP for funding to continue our work to bring immersive design and technology with this and other Cornish museums.
Impact This project is multidisciplinary: Museum Studies; Interaction and Interface Design; Computing; Digital Art (3D modelling in an Augmented context); storytelling/digital narrative. Research outputs: Conference presentations 'Towards a Blueprint for Mixed Reality Experiences in GLAM Spaces: The Augmented Telegrapher at Porthcurno Telegraph Museum', 32nd BCS Human Computer Interaction Conference, 2-6 July, 2018, Belfast, NI.; ITiCSE 2018, July 02 - 04, 2018, Larnaca, Cyprus; 'Weaving Time and Space: Augmented Reality in a Museum Context', Time, Space and Culture conference, London Centre for Interdisciplinary Research, Oxford University, Feb 15-16, 2020. 2 Research papers based on our collaborative practical work. 1. Scott, Michael James, Parker, Alcwyn, Powley, Edward, Saunders, Rob, Lee, Jenny, Herring, Phoebe, Brown, Douglas and Krzywinska, Tanya (2018) Towards an Interaction Blueprint for Mixed Reality Experiences in GLAM Spaces: The Augmented Telegrapher at Porthcurno Museum. In: Proceedings of the 32nd BCS Human Computer Interaction Conference, 2-6 July, 2018, Belfast, NI. 2. Parker, Alcwyn and Scott, Michael (2018) Crafting Engaging Programming Experiences for Young People in GLAM Spaces: The iOi-Sphere. In: ITiCSE 2018, July 02 - 04, 2018, Larnaca, Cyprus. We also have one paper currently in review, 'From Immersion's Bleeding Edge to the AugmentedTelegrapher: A Method for Creating Mixed Reality Games for Museum and Heritage Context' with the Journal of Computing and Cultural Heritage. Expected acceptance notice is late January 2020. Cultura outcomes and impact: - Creation of a foundational model and the development of a tried and tested workflow pattern that helps immersive developers/researchers and museums to work together that accommodates different stakeholders, priorities and those between both practice and language. - Exemplar demonstration to small museums of the possibilities of using immersive technologies as interpretational and engagement tools. - Immersive technology and design exemplars for those working within and with the Heritage sector. - Using emergent technology to facilitate engagement with harder to reach teenage audiences with heritage. - Extension of the Museum's offer - a highly marketable addition - and increased interpretational capability. - Rural museum utilises new 'bleeding edge' technology that is not yet domestically accessible in keeping with the tech innovation theme of the location. Knowledge Exchange: By virtue of the collaboration the Museum has built new technology into its future plans and we have won funding to do this Museum staff engage with new technologies, their use and management in the museum context, extending skills and knowledge. Future Economic impact: once the Escape Room experience has been fully tested polished and implemented on-site; subsequent work using beacon technology will help to attract visitors from the surrounding area.
Start Year 2017
 
Title The Augmented Telegrapher 
Description A framework for syncing multiple Microsoft HoloLens devices with an environment and interacting with tangible interfaces within the environment. This framework can be used to develop and deploy multi-player mixed-reality experiences for Microsoft HoloLens, and has been tested in a museum context. 
Type Of Technology Software 
Year Produced 2018 
Open Source License? Yes  
Impact This software has been provisionally deployed as a multiplayer immersive experience in a museum context, and forms the foundation for further commercialization. 
URL https://github.com/Falmouth-Games-Academy/AugmentedTelegrapher
 
Description Demonstration of Special Interest Group (Porthmeor Art Studios) 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact 150 people attend Porthmeor Studios as a launch for the South West Creative Technology network funded digital residencies, organised by Falmouth University. Visitor received a talk about the Augmented Telegrapher project and they had a hand-on demonstration of the prototype that we had built. Those undertaking to 'having a go' at the haptics and hololens-based experience we had developed expressed in many cases delight with their experience. Many were completely new to the technology and many traditional artists were among the attendees, who found the capabilities of augmented in this format surprising and exciting '.
Year(s) Of Engagement Activity 2019
URL https://swctn.org.uk/2019/05/16/saturday-18th-may-launch-of-the-swctn-porthmeor-residencies-and-cele...
 
Description Isles of Scilly Symposium 7-10 May 2019. Part of the Creative Scilly Festival 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Industry/Business
Results and Impact 50 people from the Isles of Scilly arts culture and heritage community as well as the IoS tourist board members attend this symposium where we presented our research and demonstrated our experimental design. This symposium is an exploration of the arts, culture and heritage in a rural or remote context. Through a series of seminars, panel discussions and studio tours, creative practitioners and organisations are invited to consider issues around strategic & professional development, best practice and partnership working for island and mainland-based creatives. This symposium is delivered by the partnership of Cultivator, Creative Islands and the University of Plymouth. The demo and talk generated a great deal of interest from participants who were eager to understand how immersive technologies and design could be used to excite visitors and extend the tourist experience.
Year(s) Of Engagement Activity 2019
URL https://www.thisisscillynews.com/single-post/2019/03/08/Programme-for-first-Creative-Scilly-festival...
 
Description Opportunities in Immersive Technology, Knowledge Exchange event. 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Industry/Business
Results and Impact Around 90 people attended this event organised by Innovate UK's The Knowledge Transfer Network on 15/03/2019. The event was designed to support immersive and digital companies and for them to learn more about the Innovate UK 'family' and regional organisations that have opportunities for supporting local and regional businesses. Regional Universities presented their work alongside regional businesses working in the immersion space. We presented a paper on our work, outlining its innovations and how we'd overcome technical issues with design solutions. In addition, we demoed a few of the haptic-HoloLens puzzles. The haptic interface we had developed to overcome the HoloLens clumsy and hard-to-use interface was what attracted the most interest from delegates.
Year(s) Of Engagement Activity 2019
URL https://www.eventbrite.co.uk/e/opportunities-in-immersive-technology-registration-55699221872?utm_ca...