Real-time Visuals for Performance, Gaming, Installation, and Electronic Environments

Lead Research Organisation: Northumbria University
Department Name: Fac of Arts, Design and Social Sciences

Abstract

The use of real-time visuals and visualizations are now commonplace in a variety of publicly recognized practises, including by VJs in dance clubs, by game designers in online multi-player games, by filmmakers for remixing their films as live cinema and by designers for the prototyping of larger physical products and environments. Some artists working in traditional mediums such as narrative film, have moved into the realm of live cinema. This includes British filmmakers Peter Greenaway and Mike Figgis who now work primarily in live cinema, remixing their films as performance.

Increasingly, musicians are adding live visuals to their work. Visual artists are working with sound as a medium alongside image. Computer programmers are employed to create cultural content that creates interactive relationships between different media. There is a need for researchers from these disciplines to share knowledge. A musician's mastery of time and live performance will benefit the filmmaker as she creates a non-linear interactive film. The visual artist's sense of composition and colour will be instrumental knowledge for a computer scientist as he develops software to allow new forms of game-like experience.

The network will be concerned with establishing various forms and norms for visuals as a performed and "live" medium. Traditionally visual mediums have been primarily experienced as "playback" mediums, in which the visual material is fixed in time and is played from beginning to end. Older examples of time-based mediums such as narrative film and video can be considered as representative of simple playback. Editing forms the core of the work that goes into preparing these forms, but when these editing structures have been finalized and applied to tape, film or DVD they run their course and experience no live variation in playback. Even complex menus with multiple options for playback on DVD (with or without commentary) cannot in anyway be considered genuinely real-time or performed.

Real-time visuals on the other hand require the intervention of a performer or a user. In the case of the VJ or live filmmaker, he or she chooses the video clips in real-time, selects the options for effects and determines the compositing of images and effects. Similarly in multi-player games the users can move freely in the environment forcing the game to adapt in real-time and produce new environments in response to both the predictable and unpredictable actions of the users. In both of the above cases some automatic processing occurs (i.e. digital signal processing of the incoming sound in the case of the VJ, prediction algorithms for drawing backgrounds in the case of multi-player games), but there is a live element that has some similarities to live music or theatrical performance.

The research network will ultimately focus its efforts on the ways in which the visual medium is undergoing a radical shift into a live, real-time and performance-based medium. This medium takes its cues from music, computer science, theatre and natural ancestors in film and video including such forms as montage. This transdisciplinary background will form a core of the research presented and as such will present work from researchers who exist in between the borders of several disciplines.

Planned Impact

Creative practitioners in the increasingly developing area of live video performance, ranging from abstract video to VJ culture would gain deeper understanding to theoretical underpinnings that contribute to development the field as well as historical perspectives. This comes out of a range of prior exhibitions, conferences, and symposia that attest to the increasing relevance of the field, including exhibitions like Son & Lumiere at the Pompidou Centre, and the Seeing Sound symposium organised in 2009. The discussions facilitated by this research network will contribute to linking theory and practice, and realising the potential of practice-based research, where research, conceptualisation and problematising of incipient cultural movements is informed by contemporary creative practice.

Who will benefit from this research?
i. Beneficiaries within the commercial private sector.
The video games industry has begun to stagnate with a focus on sequels and genre-based games and a lack of experimentation and development. This research will connect high-level researchers with video game industry practitioners to examine and create new methodologies and approaches for real-time gaming and interactive visualization within the realm of play. We will partner with existing conference organizers within the gaming community, specifically the GameHorizon conference and members of Codeworks Connect.

Except for a handful of notable exceptions, filmmakers have failed to capitalize on the potential of interactivity. At the CameraImage Festival in 2005 Mike Figgis noted that: Filmmaking has become "boring and perhaps need[ed] to become even worse before anything better can emerge." At the turn of the century Figgis transitioned into digital film and is at the forefront of performative filmmaking. The network will allow filmmakers to explore new regions of filmic time and language creating synergies with programmers VJs and game developers.

ii. Beneficiaries, including policy-makers, within international, national, local or devolved government and government agencies.

With the arrival of live audiovisual performance in popular culture, and its emphasis on digital media technologies, there is significant interest in this area in the cultural sector and the creative industries.

The Arts Council has established a policy for broadening audience participation under the banner, "Great Art for Everyone". The "Digital Opportunity" has been seen as a key driver in this. The restructuring of the Arts Council and the recent introduction of National Portfolio Organisations create forms of consolidation where both policymakers and fundees will increasingly be working across disciplinary boundaries in tapping contemporary forms of creative practice. The knowledge generated by the proposed research network feeds directly into this dynamic, contributing to policy, and benefitting NPO's.

iii. Beneficiaries within the public sector.

Beneficiaries will include the general public, researchers, curators, artists, academics, students, performers, critics, theorists, commercial and cultural organisations in the UK and internationally. Outputs from the workshops including papers, reports, publications and new partnerships leading to the creation of new cultural artifacts will be of benefit to relevant cultural institutions, production companies and arts festivals in particular.

iv. Beneficiaries within the wider public.

The audience at the public events would be exposed to exciting and dynamic new definitions of interactive performance, installation and games.

Given the experimental nature of some of the work being done in this area it is difficult to predict the exact time-line of when these benefits will be realized; however, the workshops will expose academics and professionals to new areas such as generative programming and this will allow them to incorporate these skills into their research projects.

Publications

10 25 50
publication icon
Aceti, L. (2013) When Moving Images Become Alive! in Leonardo Electronic Almanac

publication icon
Correia, N. N. (2013) AVVX: A Vector Graphics Tool For Audiovisual Performances in Leonardo Electronic Almanac

publication icon
Gibson, S. (2013) Simulating Synaesthesia in Real-time Performance in Leonardo Electronic Almanac

publication icon
McCarthy, L. (2013) Gathering Audience Feedback On An Audiovisual Performance in Leonardo Electronic Almanac

publication icon
Richardson, P. (2013) A 'Real Time Image Conductor' Or A Kind Of Cinema?: Towards Live Visual Effects in Leonardo Electronic Almanac

publication icon
Ritter, D. (2013) A Flexible Approach For Synchronizing Video With Live Music in Leonardo Electronic Almanac

 
Title Real-time Visuals 
Description This is a documentation video of our final event at the Real-time Visuals conference. It included a variety of performances from game-based audio to interactive theatre to live cinema performance. 
Type Of Art Artistic/Creative Exhibition 
Year Produced 2014 
Impact We are now working on a partnership with one of our main performers, well-known London-based artist/designers The Light Surgeons." 
URL https://vimeo.com/87866845
 
Description We have developed a robust network of researchers, technologists and practitioners interested in the use of visuals in a live context. This was developed over three workshops and a conference. The workshops were held at Northumbria University, Goldsmiths University of London, Tyneside Cinema Newcastle. The conference was held at Northumbria University, Tyneside Cinema Newcastle and Culture Lab Newcastle University.

The findings of the team were varied, but they focused generally around the historical and theoretical implications of working in time-based live visuals.
Exploitation Route These findings will be useful to any researcher or practitioner interested in real-time performed visuals. The volume in the URL above is the first published critical exploration of this topic.
Sectors Creative Economy,Digital/Communication/Information Technologies (including Software),Culture, Heritage, Museums and Collections

URL http://www.leoalmanac.org/vol19-no3-live-visuals/
 
Description The workshops and conference were well-attended by researchers, students and the general public. This had a clear impact by virtue of the public nature of these events. Summaries of these are available online at http://www.realtimevisuals.org/ Also 13 of our network attendees published papers in our Leonardo Electronic Alamanac: http://www.leoalmanac.org/vol19-no3-live-visuals/
First Year Of Impact 2013
Sector Creative Economy,Digital/Communication/Information Technologies (including Software),Culture, Heritage, Museums and Collections
Impact Types Cultural

 
Description Real-time Visuals Workshop 1 (Northumbria University, Culture Lab Newcastle University) 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Participants in your research and patient groups
Results and Impact Around 30 participants (including students and employees from local design agencies) attended the day workshops. These were both theoretical and directly practical and exposed the audience to techniques and tendencies in real-time visuals. Around 50 participants attended the evening performances at Culture Lab and audience members were invited to interact with some of the systems on display.

Most participants positively responded to the event and many attended later events.
Year(s) Of Engagement Activity 2013
URL http://www.realtimevisuals.org/workshops/workshop1/
 
Description Real-time Visuals Workshop 2 (Goldsmiths University of London) 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Participants in your research and patient groups
Results and Impact Around 75 people attended the day workshop programme at Goldsmiths, which was primary theoretical in nature. The workshops exposed the audience to a number of new tendencies in real-time visuals production. Local, national and international media artists attended this workshop. Around 75 people attended the evening performances.

Audience reaction was positive and there was international impact given the high-profile nature of some of the presenters.
Year(s) Of Engagement Activity 2013
URL http://www.realtimevisuals.org/workshops/workshop2/
 
Description Real-time Visuals Workshop 3 (Tyneside Cinema) 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Public/other audiences
Results and Impact This workshop and performance evening took place in a public venue, Tyneside Cinema in Newcastle. Therefore the audience included a large portion of the more general public. The day programme was limited to 30 due to the venue size. This allowed for an engaged interaction between the workshop presenters and the public. The evening performances took place in a large cinema and attracted an audience of approx. 75 people. Audience were able to directly feedback on one of the works presented and engagement was very high.

The participation of Tyneside led to further collaborations between the cinema and Northumbria University, including the participation of Tyynside in our final conference.
Year(s) Of Engagement Activity 2014
URL http://www.realtimevisuals.org/workshops/workshop-3/
 
Description Real-time Visuals conference (Northumbria University, Tyneside Cinema, Culture Lab Newcastle University) 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact Researchers, technologists, artists, students and the general public attended talks, workshops, and performances. The audience numbers ranged from 30 at Tyneside Cinema (due to limited seating in the venue this was the maximum) to over 100 at the Light Surgeons performance at Culture Lab Newcastle University.

The conference received very positive reviews in our Survey Monkey poll of the event. A further bid is planned involving key participants.
Year(s) Of Engagement Activity 2014
URL http://www.realtimevisuals.org/conference/