Cinema Unframed: exploring the screen in virtual reality, through live installation and mobile app

Lead Research Organisation: University of Kent
Department Name: Sch of Arts

Abstract

This project will adapt and combine key research outputs from the Digital Transformations project 'The Audiovisual Essay: a digital methodology for film and media studies' into an 18-24 minute interactive 'VR essay film', comprising three episodes that explore the foundational role of framing, editing, and sound within the film-making process. Each episode will allow users to explore and think through one of the key paradigm shifts associated with rise of VR: the transformation of the letterbox frame into 360 degree space, the abandonment of continuity editing and montage, and the dissolution of the traditional 'sound mix' into an interactive binaural soundscape.

The original project explored the use of digital video-making as a methodology for conducting research in the fields of film and media studies. In particular, it tested different ways in which the 'audiovisual essay' (or 'video essay') could complement traditional written scholarship on film and media to engage with digitally native audiences. The rise of VR over the last eighteen months opens radically new opportunities for the video essay to do so. This project explores the possibility of channeling the immersive power of VR for critical and educational ends, and in particular of using VR as a tool for communicating research on film and media to new audiences. By creating a VR essay film, the project will make possible a range of active dissemination and public engagement activities in collaboration with BFI Education. These will significantly augmenting the impact of the original project across three key groups of beneficiaries: new 18-30 year old cinemagoers, international VR creators, and the (so far latent) UK-based VR production sector. In doing so, this project follows Mark Zuckerberg's $2 billion gambit that VR provides one of the keys the future of screen media.

Planned Impact

The project will capitalise on the intense current interest in VR, both among consumers and within the creative industries, to significantly escalate the impact of key research outputs from the original project. It will do so through four pathways to impact:

i) International knowledge exchange through collaborative VR production
In September 2015, Disney invested $66m in US-based VR production startup Jaunt. In the 18 months since, Jaunt has become a leading global content producer, while the UK's VR industry remains latent. This project will form a small step towards the development of an indigenous UK virtual reality industry. It will do so by bringing leading international VR agency Vrtov into a collaborative UK-based production. The resulting knowledge exchange will then be passed on within the UK through subsequent VR productions. In tandem, the PI will collaborate with BFI Education to organise a public event to encourage conversation and collaboration between UK creatives engaged in or interested in VR production. Beneficiaries of this pathway to impact will include the film-makers themselves, the UK crew employed on their project, VR-curious creatives, and the UK's future VR sector.

ii) International exhibition as a live installation at major industry-focused film festivals.
The VR film will be installed at numerous industry-focused, international film festivals. In tandem with this festival exposure, the PI and project partners will collaborate on a publicity campaign targeted at the creative industries - through panels and workshops at festivals, articles and interviews in industry publications and websites, and practitioner-focused blogs and podcasts. The primary beneficiary of this pathway to impact will be the emergent international VR production community. Through the knowledge exchange that results from the VR film's international release, the project will make a direct intervention into the present and future of VR film-making.

iii) Collaboration with the British Film Institute on a live installation at BFI Southbank and subsequent educational activities.
Transforming research outputs from the original project into a live VR installation at the BFI will provide a compelling vehicle for audiences to engage with film and media research. In particular, the PI will collaborate with BFI Education to ensure that the project will feed directly into the BFI's current five-year plan, which focuses on engaging with new 16-30 year-old audiences. This will be done by integrating the live VR experience into BFI Education's programme of activities - for example, by organising school visits around the installation for Key Stages 4 and 5, by providing accompanying class plans, and by targeting one session of the public event at young creators.

iv) International release via free downloadable apps, viewable through easily accessible and low-cost headgear.
Each of the above activities will be carefully timed to provide a springboard to the next. As the VR film progresses through its various exhibition platforms, its public profile will cumulatively increase. The project will capitalise on this building momentum by actively disseminating the VR film online immediately after its exhibition at BFI Southbank. By utilizing popular VR aggregator apps (for example Within, Arte VR, and Kaleidoscope), this final pathway to impact will focus on extending the benefits of the project to as wide range of domestic VR users as possible.

Publications

10 25 50
 
Title A Machine for Viewing 
Description A three-episode, 18-24 minute, 'interactive video essay', for viewing as part of a real-time virtual reality installation through Oculus Rift headsets, and in the form of of a live 'mixed reality' performance by the work's three creators: Richard Misek, Charlie Shackleton, and Oscar Raby. 
Type Of Art Artefact (including digital) 
Year Produced 2019 
Impact 'A Machine for Viewing' received its world premiere at the International Documentary Festival of Amsterdam (IDFA) in November 2019, where it also formed the basis of a presentation and round table discussion at the annual immersive industry DocLab conference at IDFA. It received its North American premiere in January 2020 at Sundance Film Festival. It has since been reviewed by Filmmaker Magazine, resulted in requests for interviews (to be published), and provided a springboard for discussions with festival programmers for installation/screening at subsequent film festivals in 2020. It will also form the basis of a short tour of UK cinemas in summer 2020. 
URL https://sundancefestival.net/a-machine-for-viewing/
 
Title A Machine for Viewing - online performance for Melbourne International Film Festival 
Description The PI collaborated with the two collaborators from his AHRC project 'Cinema Unframed: exploring the screen in virtual reality, through live installation and mobile app' (AH/R004854/1) to create a 'lockdown' version of their live VR experience 'A Machine for Viewing' for Melbourne International Film Festival ( 12th March, 2021). The work was a hybrid of interactive Zoom performance and live filming at an empty art deco cinema, combined into a YouTube livestream. The work comprised a self-contained creative research work package into the the use of Zoom as a platform for interactive performance. 
Type Of Art Artistic/Creative Exhibition 
Year Produced 2021 
Impact The performance was the first 'digitally native' commission for 'MIFF Play', Melbourne International Film Festivals's new online portal. Its success opened the door for future digitally-native arts commissions by the festival in 2022. 
URL https://if.com.au/miff-announces-launch-of-a-machine-for-viewing/
 
Description The grant has led to the development of the first ever video essays to utilise fully-interactive virtual reality technology. The video essay is a now well-established mode of engaging critically with film and media, typically comprising montages of found footage combined with voice-over commentary. This project has extended the boundaries of the form, through the creation of a three-episode work entitled 'A Machine for Viewing' that makes possible various haptic interactions with film clips. Positioning the user in a virtual cinema, each episode uses a particular interaction as a tool for facilitating critical engagement with film. Episode 1 ('A Frame of the Mind' by Charlie Shackleton) guides users through the historically changeable shape of the cinematic frame by enabling them to manipulate the aspect ratios of movies from Sergei Eisenstein's Battleship Potemkin to Xavier Dolan's Mommy. Episode 2 ('A Pillow of Light' by Richard Misek) enables the user to touch and cross 'into' the cinema screen; it uses this interaction as a springboard for exploring how cinematic immersion and the haptic immersion of VR

how the immersive impact of the cinema screen is influenced by its size and proximity to the viewer. Episode 3 ('Manual for a Disassembly of Cinema' by Oscar Raby/Vrtov) places the user in the projection booth; by guiding them through the process of troubleshooting a malfunctioning Barco projector, it explores how information technology shapes contemporary cinema exhibition.

Various discoveries have emerged from the project's innovative combination of conventional film, 360-degree video, and 'real-time' virtual reality. For example, the project:
- Highlights the ontological distinction between, and the potential for hybridisation of, 360-degree video and computer-generated 3D virtual reality environments,
- Demonstrates the added value of incorporating real-time responsiveness and haptic engagement into a digital learning environment, and
- Proposes a range of techniques for combining linear narrative (in particular, verbal narration) and game-inspired interactions

Further discoveries will emerge following the installation of A Machine for Viewing at international film festivals and a tour of UK cinemas in 2019.
Exploitation Route By combining the narrative and analytical focus of the video essay with the immersive experience of virtual reality, the project opens up new possibilities for both the video essay and nonfiction VR. It advances the former by providing new interactive tools for exploring and interrogating film and media; it advances the latter by demonstrating how immersive media can be used to deliver (narrative) nonfiction content. It is therefore anticipated that the project's findings will be of particular interest to the following:

1. VR creators working at the intersection of gaming and nonfiction narrative.
The project will provide a model for 'VR documentary' creators aiming to combine story-based narratives with user agency. For example, the discoveries made by the project's main partner, VR studio Vrtov, are already feeding into subsequent practice through Vrtov's contribution as project partner to the current EPSRC research project, 'Virtual Realities: Immersive Documentary Encounters'.

And

2. Academics engaging in audiovisual and practice-based film scholarship.
The project will provide scholars engaged in audiovisual film and media studies with a model for how to move beyond the montage-based tropes of traditional video essaying. It will also form a basis for future experiments with the video essay form: following A Machine for Viewing's festival exhibition, the Unity project in which it was constructed will be made open access, allowing future audiovisual scholars to use the project's assets and code to generate new works.
Sectors Creative Economy,Culture, Heritage, Museums and Collections

URL https://www.sundance.org/projects/a-machine-for-viewing
 
Description The main output of the project, the real-time VR video essay 'A Machine for Viewing' (co-authored with project partners Charlie Shackleton and Oscar Raby/Vrtov) premiered in 2019 at the International Documentary Festival of Amsterdam (IDFA), the largest and most prestigious documentary film festival in the world. It was demonstrated to an audience of 300 leading figures in the immersive entertainment sector at the annual IDFA DocLab conference, and formed the basis of a roundtable discussion about the relationship between film and immersive media. The roundtable brought together academics from the MIT Open Documentary Lab, industry practitioners, and industry decision-makers. 'A Machine for Viewing' was subsequently screened at the festival to a public audience of 300, in the form of a mixed-reality performance based around the core VR experience built as a part of the project. As one of only five live performances presented over the two weeks of IDFA, it formed a key offering by the festival, and provided an opportunity for both industry practitioners and the general public to engage with questions around the future of cinema and its relationship with emergent media. As well as contributing to industry discourse and helping frame its audience's critical engagement with emergent media, the project also had an impact on its creators. The collaborative process involved all three creators having to speak across the disciplinary divide between filmmaking and immersive media; the transdisciplinary nature of the production process demanded that the filmmakers (Charlie Shackleton and myself) learn to use Unity 3D and, more broadly, engage with a form in which narrative is presented spatially, and not as a linear temporal flow; and it allowed us to bring the insights learnt from this process back to our subsequent filmmaking process. For example, the experience of working on the project inspired filmmaker Charlie Shackleton to create a subsequent VR video essay / documentary / live performance entitled 'As Mine Exactly' (2022); this was performed at True/False Film Festival and the Museum of Moving Image (New York) in March 2022. The collaboration also provided an opportunity for the project's creative technologist Oscar Raby to explore what filmmaking paradigms could be transferred to his pioneering work as a VR artist, and provided him with new conceptual tools for combining linear narrative with interactivity. 'A Machine for Viewing' had its North American premiere at Sundance Film Festival, where it similarly played a role in shaping industry discourse about interactive narrative, and provided the general public (total audience of 500, over two nights) with a platform for engaging with key issues around transmedia and media hybridity. Its contribution to industry discourse was reinforced by two articles in Filmmaker Magazine that used it as a way in to engaging with the current state of immersive media, as well as a podcast for the publication 'VR User'. Documentation of (and an introduction to) 'A Machine for Viewing' was published in NECSUS (the journal of the European Network of Cinema and Media Studies) in summer 2020, through which it broadened its international reach beyond the film festival circuit. This non-interactive iteration of the work was cited by multiple critics in Sight and Sound magazine's 'Best Video Essays of 2020' poll. The work was due to tour a number of major UK cultural venues including Home in Manchester and FACT in Liverpool in late summer 2020, as well as screening at further festivals, but this route to impact was cut short by COVID-19. The creators of the work did, however, reconfigure it as a performance livestreamed on YouTube from an empty cinema for Melbourne International Film Festival in March 2021.
First Year Of Impact 2019
Sector Creative Economy,Electronics,Culture, Heritage, Museums and Collections
Impact Types Cultural

 
Description A screening of 'A Machine for Viewing' at Sundance Film Festival 
Form Of Engagement Activity A broadcast e.g. TV/radio/film/podcast (other than news/press)
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact A screening and live performance 'A Machine for Viewing' was delivered over two nights at Sundance 2020, as a part of their New Horizon programme. The screening resulted in a review (next entry), interviews (to be published), and direct user responses via social media (e.g. tweet from Raqi Syed - 'A MACHINE FOR VIEWING is the meta, immersive, & accessible version of every essay about expanded cinema I have ever written or attempted to write. If you get a chance to experience this film-do it! #Sundance2020 #sundanceNewFrontier #futureOfCinema').
Year(s) Of Engagement Activity 2020
URL https://sundancefestival.net/a-machine-for-viewing/
 
Description Article on 'A Machine for Viewing' in Filmmaker Magazine 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact Two separate articles focusing on 'A Machine for Viewing' in Filmmaker Magazine. One was an in-depth review, the other was a round-up of the entire New Horizon programme at Sundance 2020 that used 'A Machine for Viewing' as the starting-point and agenda-setter for discussion on the current state of immersive film and virtual reality.
Year(s) Of Engagement Activity 2020
URL https://filmmakermagazine.com/109210-looking-for-the-new-at-the-2020-sundance-new-frontier/#.Xl0n8i2...
 
Description Film Festival Screening (International Documentary Festival of Amsterdam) 
Form Of Engagement Activity A broadcast e.g. TV/radio/film/podcast (other than news/press)
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact An audience of 300 attended the world premiere of 'A Machine for Viewing' at the International Documentary Festival of Amsterdam (IDFA), of whom about 250 stayed on for an extended interview with the PI and his collaborators.
Year(s) Of Engagement Activity 2019
URL https://www.idfa.nl/en/film/3984bb8e-6309-4f06-bdaf-609736be63cd/a-machine-for-viewing
 
Description Presentation at DocLab Interactive Conference, International Documentary Festival of Amsterdam (IDFA) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact 250 leading figures in the immersive / interactive film sector attended a presentation on immersive nonfiction by the PI and his collaborators (Charlie Shackleton and Oscar Raby) at the annual industry 'DocLab' symposium at the International Documentary Festival of Amsterdam. The PI and his collaborators also took part in a concluding round table at the end of the day. The aim of the presentation was to influence the current discourse around how VR can be utilised for narrative purposes, by comm
Year(s) Of Engagement Activity 2019
URL https://www.idfa.nl/nl/shows/4205fe9f-1a2f-479e-97d8-79594f19ae85/doclab-interactive-conference#tick...