Immersive Pipeline: Production pipelines and translators for the authoring, sharing, and touring of immersive media performance works

Lead Research Organisation: Goldsmiths University of London
Department Name: Computing Department

Abstract

Immersive media and virtual reality are not the same thing. While VR brings with it the image of experiencing virtual worlds through 3D headset displays, the actual use of such displays can be physically disorienting and socially isolating. Immersive audiovisual experiences can be created in public space and in performance by the seamless projection of data, graphics, and video on all surfaces of a performance space, and by the use of multichannel surround sound. Immersion can take place in real space and be shared with others as an embodied collective experience. Goldsmiths' Sonics Immersive Media Labs (SIML) is part of an international network of immersive projection venues including rectangular surround video facilities like the RML CineChamber (San Francisco), domes such as the SATosphere (Montreal), or new formats such as the ISM Hexadome (Berlin). Whilst these facilities exist around the world, there has yet to be established a standard working method for the creation of new cultural works for them. Artists find themselves dedicating an inordinate proportion of their creative time understanding how to configure their work to fit an immersive space, often reinventing the wheel. Once a work has premiered, they find it difficult to tour it out of lack of compatibility across these venues, resulting in inadvertently site-specific work.

We are experiencing a teething period in immersive media not dissimilar to the early days of cinema, before a lingua franca of authoring method and technical process was established. The Immersive Pipeline project will partner with these leading cultural sector institutions above, as well as with immersive technology companies Derivative and NSC Creative, to form a partnership to design and develop an artist-facing immersive media production workflow. The project will engage with artists, scientists, presentation venues, and technology companies in participatory design sessions to define best practice for creative immersive media production. We will produce an immersive content authoring pipeline and make it available to artists as a method and process with which to create new works for multiscreen spaces. By partnering with the Elizabeth Garrett Anderson Institute for Women's Health, the project will demonstrate the pertinence of artistic immersive visualization and sonification in the health and medical sector.

Finally, we will work with our partner venues to define the specification for, and prototype, a translation and format conversion technology allowing work created for one venue (for example a dome) to be automatically translated for projection in another (a rectangular venue). The results of the project will:
1.) Empower artists to produce works for a range of immersive venues,
2.) Advance the state of the art of projection mapping technology,
3.) Bring together a diverse group of cultural institutions through shared vision and technical interoperability, and
4.) Bootstrap an international consortium for future large research projects.
The results of the Immersive Pipeline project will be showcased in a public performance/screening event.

Planned Impact

As a pilot project, the impact of the 6 month project will be preliminary and indicative. As the artist and cultural venue beneficiaries described above gain the knowledge to produce and present work for immersive spaces, the potential for broader impact on industry and culture will become apparent. This includes impact in mainstream cinema, responding to interest expressed by large organisations like the British Film Institute (BFI) and industry companies like IMAX (https://www.wired.com/2017/01/imax-vr-theaters/). This has the potential to trigger a sequence of impact from practitioner and industry bodies leading to influencing policy at the governmental level. This upward arch starts with Immerse UK (www.immerseuk.org), followed by the Creative Industries Federation (https://www.creativeindustriesfederation.com/) to future policy briefs following on from NESTA's 2016 report, "The Geography of Creativity in the UK" and fulfillingthe creative economy elements in HM Government's 2017 green paper, "Building our Industrial Strategy."

Our work with the EGA Institute for Women's Health's Sonic Womb project will have an impact in society beyond the academic benefits described in the previous section, and point out the potential for immersive audiovisual projection to aid in the public communication of science leading to greater public understanding of health and well being.

A specific process of identifying and fulfilling impact is described in the Pathways to Impact section.

Publications

10 25 50
 
Title Cave Music, with Uta Kogelsberger 
Description Cave Music was the 3rd incarnation of this 8 channel sound installation of free standing, simulated "rock-speakers" spreading mixes of Kögelsberger's recordings captured in the mouth of a disused mine, creating a trancelike, rhythmic and ritualistic soundscape hovering between abstraction and dance track. Cave Music - a playful evocation of man's often futile attempt to control nature - took its cue from the increase of rainfall in the Lake District due to climate change. It followed on from this video installation Playing the Cave and Orchestra of Rocks not a Rock Concert. During an evening of live performances, experimental sound artists Atau Tanaka, Shelley Parker, Dane Law, and Chooc Ly created their interpretations of the original sounds. In the Battle of the Rocks, devised by Kögelsberger, the beat boxing champion Marv Radio challenged the Rocks to a battle between voice and environmental sound. 
Type Of Art Artistic/Creative Exhibition 
Year Produced 2019 
Impact Working with community arts organisation, Ugly Duck. http://uglyduck.org.uk/portfolio_page/rockconcert/ 
URL https://utakogelsberger.net/rock-concert-cave-music/
 
Title Gone Gone Beyond, by People Like Us 
Description Gone, Gone Beyond is an immersive a/v spatial cinema work by People Like Us, which breaks the rectangle, smashing the thin screen into tiny fragments, looking beyond the frame, climbing through to see what's behind. Commissioned by Naut Humon, the founder of immersive theatre project CineChamber, Gone, Gone Beyond is a 10 screen/8 speaker work by Vicki Bennett with seamless wrap around projection and surround sound, where the audience sit inside. It uses edited collage sewn together in a giant patchwork. Pull on a thread and watch whole new narratives expand and unravel all at once on a 360 palette. The project has been a work in progress since 2017, and is currently 50 minutes long. History: the initial in-process tester movie screened in San Francisco in October 2017 at RML's own Recombinant Festival at Gray Area Foundation for the Arts. Since then we have been continuing to work on content, and staged a private event in April 2019 at Goldsmiths SIML to encourage some partner support in the UK/Europe. 
Type Of Art Film/Video/Animation 
Year Produced 2019 
Impact Working with the cultural sector: Sound & Music in the UK, and the Recombinant Media Labs in San Francisco USA 
URL http://peoplelikeus.org/2019/gone-gone-beyond/
 
Title Le Loup, Lifting, and Myogram 
Description A gestural sound/image performance by Atau Tanaka and visual artist Lillevan 
Type Of Art Performance (Music, Dance, Drama, etc) 
Year Produced 2018 
Impact This was performed at the IX Symposium in Montreal and in SIML Goldsmiths 
URL https://ix.sat.qc.ca/node/879?language=en
 
Title Rock Concert II with Uta Kogelsberger 
Description Loosely based on the notion of a ritualistic "stop the rain dance" Rock Concert II is a visceral response to the increase of rainfall in the Lake District due to climate change. Recordings of individual raindrops captured by Uta Kögelsberger during heavy rainfall in the mouth of a disused mine, with experimental recording techniques are handed over to sound artists including Atau Tanaka, Will Edmondes, The Noise Choir and Mariam Rezaei. An 8 channel sound installation of free standing, simulated "rock-speakers" spread live mixes creating trancelike, rhythmic and ritualistic soundscapes hovering between abstraction and dance track. The work takes its cue from the previous video installation 'Playing the Cave'. 
Type Of Art Artistic/Creative Exhibition 
Year Produced 2018 
Impact Taking the work to northeast England. Working with local artists. 
URL https://utakogelsberger.net/rock-concert-ii/
 
Title Skin Awareness II by Youhong "Friendred" Peng 
Description Dancer's body communicates with the immersive space extending the dancer's sensory. Skin-awareness explore the relationship between the body and space, and experiment with the capacity of the body while interfering with the digital technologies that change the surrounding space. The body is not only an entity, but also an action, an experience and an embodied perception. The immersive space morphs and alters with light and becomes solid, its pressure composing and decomposing the self-awareness of skin. The dancer's body is extended and manipulated as a conscious entity, exceeding the physiological object. In Skin-awareness, the embodied, symbiotic relationship of light, sound, and space is reciprocal. This work crosses the intersection of technological interventions, architectural fabrication, and contemporary dance performance. Forty laser beams controlled by mechanical robots read and encode choreography. Synergies of futuristic lighting and surrounding screens create a brief detachment from reality. Computer vision is utilized with computational precision along with the control of body movement. 
Type Of Art Performance (Music, Dance, Drama, etc) 
Year Produced 2019 
Impact Working with community arts organisation, Ugly Duck http://uglyduck.org.uk/portfolio_page/skin/ 
URL http://friendred.studio/2019/04/28/skin-awareness-london-ugly-duck/
 
Description There is no universal immersive format, it depends on the nature of the artists' project. Artists need access to facilities.
Exploitation Route Our tools and methods are openly available to communities of practitioners.
Sectors Creative Economy,Culture, Heritage, Museums and Collections

URL http://sonics.goldsmithsdigital.com/immersive-pipeline/
 
Description We created tools for immersive audiovisual performance that are used by artists performing internationally. The artists, People Like Us were able to use our immersive facilities at Goldsmiths to prepare their international US premier of Gone Gone Beyond in San Francisco USA at the Recombinant Festival in October 2019. This award enabled a collaboration with the music charity, Sound & Music, to support People Like Us' "New Voices" residency. http://peoplelikeus.org/tag/gone-gone-beyond/
First Year Of Impact 2019
Sector Creative Economy,Culture, Heritage, Museums and Collections
Impact Types Cultural

 
Description Immersive Pipeline 2: Out of the lab and into the wild
Amount £32,099 (GBP)
Funding ID AH/S010637/1 
Organisation Arts & Humanities Research Council (AHRC) 
Sector Public
Country United Kingdom
Start 01/2019 
End 12/2019
 
Title Immersive Pipeline whitepaper 
Description This report examines immersive experiences in the context of social shared spaces. It presents the Immersive Pipeline research project led by Prof. Atau Tanaka and funded by the AHRC/EPSRC Research and Partnership Development call for the Next Generation of Immersive Experiences from January to June 2018. It covers a historical and technological overview of state of the art on this field, cases of study and interviews in depth with some of the participants. 
Type Of Material Improvements to research infrastructure 
Year Produced 2019 
Provided To Others? Yes  
Impact making available to other audiovisual and immersive practitioners 
URL http://research.gold.ac.uk/id/eprint/28259
 
Description Immersive Pipeline at Splice Festival 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Public/other audiences
Results and Impact We presented the Immersive Pipeline project at the Splice Festival
Year(s) Of Engagement Activity 2018
URL http://www.splicefestival.com/line-up/immersive-pipeline/
 
Description Immersive Pipeline performance event 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Public/other audiences
Results and Impact Performance event from the Immersive Pipeline project in the Sonics Immersive Media Labs (SIML) at Goldsmiths
Year(s) Of Engagement Activity 2018
URL http://sonics.goldsmithsdigital.com/immersive-pipeline/