Building an international network for virtual dance collaboration: Deploying Goldsmiths MoCap Streamer tool for inclusive and sustainable development
Lead Research Organisation:
Goldsmiths University of London
Department Name: Media and Communications
Abstract
'Goldsmiths Mocap Streamer' is a software tool that was initially developed to address the issues surrounding the isolation and physical distancing of the Covid19 restrictions, by first connecting dancers remotely through a framework for motion-capture data streaming, and then seeing what kinds of emotional, aesthetic, and affective connections could be made within virtual spaces. These concerns were framed and led by our collaborating artists, honed and developed through a series of practice-led research workshops with artists both local to Goldsmiths in London, and as far away as Singapore (LASALLE) and Hong Kong (HKAPA). However, we knew that beyond the immediate concerns of the pandemic, this tool had much broader applicability, possibly offering itself as a critical turning point in a 50-year history of the aesthetic and technological evolution of telematic performance practice. Having now reached a point where we have evidenced the functionality of our Mocap Streamer framework in both technical and aesthetic-poetic terms, we now wish to roll it out to a truly international and diverse network of dance companies as a next logical step in our research process. We aim to empower performers who might not have previously considered doing digital dance work to freely experiment in the forms of aesthetic expression and meaningful communication that we have already shown are possible.
We propose that this research would have three overarching priorities:
1. A connector programme using a bursary/residency structure, to bring together small dance companies and creative technologists from diverse international backgrounds for six full months of collaborative work. We will provide them with a mocap starter toolkit and a creation/development budget, and will facilitate them to develop their own, and collaborative, digital work. We adopt a user-centred and research through design methodology - creating practice-led feedback loops to develop our framework for remote dance collaboration.
2. To continue to develop and hone our existing motion capture data streaming tool in terms of efficiency, accessibility, and user experience (UX), incorporating the creation and development of a new AI tool for motion capture system interoperability - essentially a Machine Learning algorithm that can recognise and share choreographic movement from many different motion capture systems in a truly system-agnostic mode.
3. To document and evidence our findings through a knowledge-sharing and performance showcase event, a dedicated website, and to develop a resource of open-source licensed toolkits and software.
Overall, our creative practice-led research methodology and programme of training, mentoring, and support will lead to a deeper understanding of the kinds of dance work that can be productively and economically achieved within this international network. We aim to open doors for several dance companies, to facilitate them to create new aesthetic approaches to dance work, and so that they can develop new audiences and new revenue streams, at the same time offering a credible and exciting use-case for international mocap streaming-based collaboration. It will form the basis for new ways of thinking about and working with motion-capture in dance, towards sustainable future development.
We propose that this research would have three overarching priorities:
1. A connector programme using a bursary/residency structure, to bring together small dance companies and creative technologists from diverse international backgrounds for six full months of collaborative work. We will provide them with a mocap starter toolkit and a creation/development budget, and will facilitate them to develop their own, and collaborative, digital work. We adopt a user-centred and research through design methodology - creating practice-led feedback loops to develop our framework for remote dance collaboration.
2. To continue to develop and hone our existing motion capture data streaming tool in terms of efficiency, accessibility, and user experience (UX), incorporating the creation and development of a new AI tool for motion capture system interoperability - essentially a Machine Learning algorithm that can recognise and share choreographic movement from many different motion capture systems in a truly system-agnostic mode.
3. To document and evidence our findings through a knowledge-sharing and performance showcase event, a dedicated website, and to develop a resource of open-source licensed toolkits and software.
Overall, our creative practice-led research methodology and programme of training, mentoring, and support will lead to a deeper understanding of the kinds of dance work that can be productively and economically achieved within this international network. We aim to open doors for several dance companies, to facilitate them to create new aesthetic approaches to dance work, and so that they can develop new audiences and new revenue streams, at the same time offering a credible and exciting use-case for international mocap streaming-based collaboration. It will form the basis for new ways of thinking about and working with motion-capture in dance, towards sustainable future development.
Publications
Strutt D
(2021)
Virtual relationships: the dancer and the avatar
in Theatre and Performance Design
Strutt D
(2021)
New Telematic Technologies for Remote Creation, Rehearsal and Performance of Choreographic Work
in Journal of Embodied Research
Zylinska Joanna
(2022)
The Future of Media
| Title | Global Dance Collaboration in the Metaverse - Being Human Festival 2022 |
| Description | 7 individual works created through mentorship, supervision and through the provision of tech by the research project. The project showcase event was a combination of live, immersive screen-based performance, in-person interactive performance, and interactive installation. It was presented in a hybrid in-person and live-streamed mode with remote dancers performing together within a virtual and interactive environment works included Sufee Yama & Pichet Klunchun Dance Company (Thailand) - 'Ghra?a - Vijñana' A Thai contemporary remote performance that allows the audience to access the performers' internal space to witness memories and experiences that are triggered by scents, causing chaos to their mind and body. This metaverse dance experience will reveal their fight and struggle, and how the battle with themselves ends. Ghra?a - Vijñana is Olfactory consciousness in buddhism. Sumedha Bhattacharyya (India) & Joaquina Salgado (Argentina) - 'Water Nodes' We are water and memory is what the body carries, Like information, like energy. Water is memory. Responding to the insurmountable and collective grief, loss and the heaviness of memory experienced during the afterlife of the pandemic, Water Nodes is a hybrid immersive installation /performance (live +remote) that reminisces cross-cultural, personal, maternal histories inherited/passed on in the body. By evoking family archive photographs, spiritual and mythological references, the performance considers in the quality of water a passage for memories to flow, contain and dissipate. Katie Dale-Everett (UK) & Erin Cuevas / Matt Conway (USA, California) - Playscape: How to Build A Galaxy 'Playscape: How to Build A Galaxy' is an immersive performance installation that combines dance, motion capture technology and projected celestial graphics to open up new possibilities for physical, social and digital connection. This work is made particularly for young people who experience anxiety in social interaction, but become motivated when engaging with technology, allowing them to discover new ways of sharing space. We welcome everyone and anyone to give it a try. Clarice Hilton, Neal Coghlan, Kat Hawkins and Susanna Dye in collaboration with Candoco Dance Company (UK) - 'Beyond T-Pose: challenging normative assumptions in MOCAP' This work-in-progress deconstructs mocap data away from its normative forms, reconstructing the data to create customised visual expressions and movement interactions. In a series of short vignettes, dancers will move together through a co-created vocabulary, pushing the boundaries of what self-representation, interaction and communication can be in the metaverse. This is the culmination of a series of workshops with Candoco dance company and associated dancers Kat Hawkins, Susannah Dye, Ted Wilkinson, Paulina Porwolik and Dermot Farrell. Ânima Cia. de Dança & Pedro Rocha (Brazil) - 'Weaving through the space mass' The dance performance proposes to make visible the weave of space and its manipulation by the dancers. Considering the space around the body has a certain density, when in movement, the body is not only sustained by this mass, but also manipulates and modifies it. Letta Shtohryn (Ukraine/Malta), with dancers in UK, New York & Malta - ''Lois. Monstrous encounters'. Based on factual and fictional experiences in confined underground spaces, this remote dance collaboration explores the moment when isolated life forms encounter the Primary World. The monstrous self and the monstrous others experience inner and outer transformation of their body~mind states, asking how to share experiences and relate to one another after a period of isolation? Can some experiences be explained / related to at all? And whose voices are believed and considered valid? Afro-Sul Grupo de Música e Dança & Pedro Rocha (Brazil) & Ainslee Robson (USA, California) - 'Alma Negra' Inspired by the archetypes of the female orishas Oxum, Iansã, Iemanjá and Nanã, the performance proposes an encounter between past and present, between the Pan-African roots of Afro-Sul and the experience of dance in the metaverse and confirms that the future is ancestral. Looking back is key to a better future! |
| Type Of Art | Performance (Music, Dance, Drama, etc) |
| Year Produced | 2022 |
| Impact | All artists describe the process of creating the work as 'enriching', 'reaffirming the direction I want my company to go in', 'opening up new possibilities for me' and saying 'the significance has been massive.' We have seen our collaborators go on tho other funded projects for Arts Council, doing workshops at Indian universities 'I have already reached out and conducted a motion capture workshop at Jawaharlal Nehru University in India with Masters and PhD students'. We have seen our collaborative practices be integrated into teaching in Brazil, India, and UK |
| URL | https://www.mocapstreamer.live/showcase-12-nov-2022 |
| Description | As stated in the proposal, this project's first objective was to create an international network of dance groups who were able to collaborate remotely towards shared choreographic practice in virtual spaces. In this sense the project was a great success. We evidenced that it was possible to have dance groups in different locations around the world working through the use of the custom Mocap Streamer software tool. In addition, we showed that it was possible to create a strong sense of virtual community through these practices, with the sharing of cultural references and knowledge from Brazil to LA, from Buenos Aires to New Delhi. However, this success story was not without limitations. Through an international open-call process in January and February 2022 we received approximately 160 applications. We specified that there had to be an element of introduction to digital dance and mocap practices, as we did not want experienced artists in this area applying. We also specified that we were looking for cultural and disciplinary diversity. Through a rigorous process we selected, matchmade, equipped with tech, and thus assembled five teams of dancers and creative technologists, who with training and mentorship worked to produce 5 performance pieces which were delivered in Being Human festival 2022. https://www.mocapstreamer.live/showcase-12-nov-2022 We evidenced here that profound and meaningful work could be made at a distance, through the use of motion capture and digital tools. Artist's testimony reveals that this was transformative to their practice. The limitations of the research design and execution included time difference clashes, technical interdependencies, and budget clashes which were exacerbated by the remoteness of the collaboration. We also discovered a limit to truly being able to bring remote dancers together for planning, rehearsal and performance, with many teams opting to use dancers in situ, even though they were using the remote streamer technology. It seemed too far out of many choreographers' comfort zone to truly choreograph remotely. Despite this, strong and convincing digital work was created that truly exhibited the power of the framework. One participant Sumedha noted: 'Surprisingly, it was a spiritually moving experience. There is immense potential in the somatic and emotive dimension of motion capture as a technology'. Sufee in Bangkok, Thailand described working on the idea of 'Cultural Teleportation': "Working in Mocap Streamer Residency has pushed me to think about the relationship between dancers, objects and space and how these together create a new language to communicate with international audiences in real time". Extenign this point Wahei in L.A. noted one of the huge benefits of the project included "Understanding how different perspective relates to artists' regional culture.... Stunning works". Overall, the finding were that despite difficulties inherent in all creative projects being in some way exaggerated by collaboration across timezones, technological and cultural barriers, beautiful and powerful work could be created and shared with international audiences. |
| Exploitation Route | This project has provided strong use cases around the use of motion capture, real-time graphics, and streaming tools in international and transcultural choreographic work that we think is both inspiring and useful to other artists and creative technologists in the field, or just starting out. One of the main beneficiaries of this work has of course been the (roughly 20) dancers and digital artists that we directly worked with, and their own extended networks, with lasting changes in focus and practice being observed and decribed. We have had a high level of engagement wioth the porject through both the open call and final showcase performance, so we do believe that the work has had a strong impact internationally. We have been very aware of the need to be open and honest about difficulties, and this in fact formed part of our final presentation, particularly around technological barriers to access with regard to disability and neurodiversity. This showcase has been watched by around 1300 times on Youtube. I have already engaged in extensive writing for publication, most recently for a multi-volume 'Pandemic and Beyond' series for Manchester University Press. The research is also being used in material for teaching at UG and MA level. The above academic and non-academic routes to impact will most certainly play out over time, with tangible consequences of our research emerging for others in their own work over the next year or two. |
| Sectors | Digital/Communication/Information Technologies (including Software),Leisure Activities, including Sports, Recreation and Tourism,Culture, Heritage, Museums and Collections |
| URL | https://www.mocapstreamer.live/artists-in-residence |
| Description | We feel that our research project has been contemporaneous with the work of only a few other key players (over the last two years) in establishing a decisive and significant new research area, and further has evidenced the possibility for a notable change in the quality of creative output in the field of digital performance. Most directly this impact is shown within the work of the participants within our artist-residency programme, with tangible changes in their own practice and future development anecdotally reported, but the effects of the research seem to spin out into the work of our partners, our own research team, and others within our network through new artistic projects (Figural Bodies), software (Alexander Whitley's Immersive Digital Dance Studio) and partnerships. While no immediate economic benefit is evidenced (our own showcase performance was free) we have shown the scope for large online audiences for virtual dance work that could convert into economic benefit for the artist themselves. I and other team members are now embarking on several other research projects, public exhibitions, and further applications for funding. I have recently been successful in winning a British Academy Innovation Fellowship to work with project Alexander Whitley Dance Company for one year, with a focus on accessibility and inclusivity in the area of digital dance, and we are currently also working on an application to the AHRC's own Dance Research Matters call. We are going to SXSW festival in Austin in March 2023 with work developed through this project and accepted into their XR Experience Spotlight programme, and we are applying to SIGGRAPH later in the year with AWDC's Digital Dance Studio. This will enhance our international profile, and will see us continuing to generate and share knowledge and impact about the research. |
| First Year Of Impact | 2022 |
| Sector | Digital/Communication/Information Technologies (including Software),Leisure Activities, including Sports, Recreation and Tourism,Culture, Heritage, Museums and Collections |
| Impact Types | Cultural |
| Description | POSTNOTE 669 May 2022 on "The impact of digital technology on arts and culture in the UK" |
| Geographic Reach | National |
| Policy Influence Type | Contribution to a national consultation/review |
| URL | https://post.parliament.uk/research-briefings/post-pn-0669/ |
| Description | Sufee Yama - Meta APAC Public Policy Department collaboration. |
| Geographic Reach | National |
| Policy Influence Type | Influenced training of practitioners or researchers |
| Impact | This placed our project output on a Thai national platform, recognising the use of motion capture and real-time graphics software in capturing traditional dance practices (material from this project can be see at 0.26 here https://fb.watch/iwYc7bcBf0/). This activity was designed to boost tourism in Thailand. Though no tangible figures are available, it's clear that this work has had reach and impact, with some 10k views on the video. |
| URL | https://apacpolicy.fb.com/thailand/rediscover-thailand/ |
| Description | British Academy Innovation Fellowships Scheme 2022-23 (Route A: Researcher-led) |
| Amount | £110,507 (GBP) |
| Funding ID | IF2223\230051 |
| Organisation | The British Academy |
| Sector | Academic/University |
| Country | United Kingdom |
| Start | 03/2023 |
| End | 02/2024 |
| Description | Mocap Steamer Artist Residency - Brazil |
| Organisation | Federal University of Rio Grande (FURG) |
| Country | Brazil |
| Sector | Academic/University |
| PI Contribution | Training and mentorship, induction and training sessions. Repositories of digital asset packs, resources, and training modules on Gitbook. Provision of motion capture hardware - value £6000 Bursary of £5000 |
| Collaborator Contribution | The collaboration was with the Carne Digital research group at UFGRS, working with eminent choreographers Eva Schul of Anima Dance, and Iara Deodoro of Afro Sul Dance Company. These researchers created two dance pieces that were delivered in our final showcase performance. |
| Impact | the dance works themself (decribed below) and a current dance movement archive project which is being developed in Brazil through use of the equipment and training that we provided. Alma Negra - Afro-Sul Grupo de Música e Dança & Pedro Rocha (Brazil) & Ainslee Robson (USA, California) Inspired by the archetypes of the female orishas Oxum, Iansã, Iemanjá and Nanã, the performance proposes an encounter between past and present, between the Pan-African roots of Afro-Sul and the experience of dance in the metaverse and confirms that the future is ancestral. Looking back is key to a better future! Weaving Through the Space Mass - Ânima Cia. de Dança & Pedro Rocha (Brazil) The dance performance proposes to make visible the weave of space and its manipulation by the dancers. Considering the space around the body has a certain density, when in movement, the body is not only sustained by this mass, but also manipulates and modifies it. |
| Start Year | 2022 |
| Description | November 2022 Showcase Performance for Being Human Festival - Global Dance Collaboration in the Metaverse |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Public/other audiences |
| Results and Impact | This event was the final showcase for our 5 artist residency teams and our own disability-led research project. 7 original works were presented with Q&A and discussion over a four-hour period. Two works were presented in-person with dancers from the UK. The other five were streamed from respective locations ranging from New Delhi, to New York, to Malta, to Porto Alegre in Brazil, and finally Bangkok. It was presented in association with Being Human Festival. The in-person audience was around 60 people. The international and online audience was 500 live on the day, increasing to 1300 on the youtube link over the following week. There was a high level of participation from the in-person audience, with lively debate and an opportunity to get into motion-capture and dance within one project. A short video about the day can be seen here https://www.mocapstreamer.live/showcase-12-nov-2022 |
| Year(s) Of Engagement Activity | 2022 |
| URL | https://www.youtube.com/watch?v=nGvJE4VaAAc&t=6815s |
| Description | SXSW 2022 - Panel Mocap Streaming and remote dance collaboration in VR |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Industry/Business |
| Results and Impact | This was a panel discussion at South by South West Festival 2022. This panel discussed the technical, creative and conceptual issues around distributed performance experiences in VR. The conversation focussed on the VR Mocap Streamer project, a partnership between Alexander Whitley Dance Company and the Department for Media and Communications at Goldsmiths University, London, which attempts to bring live, fully embodied interactive performance from remote locations into a multi-user VR experience based on the seminal ballet The Rite of Spring. A lively discussion followed. This has directly led to the inclusion of one of our project outputs in SXSW 2023 XR Experience Spotlight programme - on the 10th to 14th March - called Figural Bodies. We also have a panel discussion 'Dance in the Metaverse - Inclusion and accessibility'. These activities are funded by British Underground, in the Future Art and Culture programme - produced by British Underground at SxSW funded by Arts Council England with additional support in 2023 from the British Council. |
| Year(s) Of Engagement Activity | 2022 |
| URL | https://schedule.sxsw.com/2022/events/PP116279 |