Immersive, Innovative, and Interactive Experience (IIIE): A Technology Driven London-Shanghai Partnership in Mixed Realities and Performing Arts
Lead Research Organisation:
Goldsmiths University of London
Department Name: Computing Department
Abstract
This is an ambitious interdisciplinary collaborative project aiming to facilitate UK-Shanghai business in the theatre and performance space, focusing on immersive and mixed reality experiences, via curated networking and partnership building events involving creative IP companies, academics and technical partners in the UK and Shanghai, including a series of Investor Showcases to fund development of concepts/IP surfaced by project partners. Our UK-Shanghai Consortium will be led by Goldsmiths, University of London, and will include:
- leading UK arts, cultural, immersive and digital content partners
- leading Shanghai arts, tech, immersive and academic partners
The project will result in sustainable model boosting economic impact of immersive and strong legacy collaborative R&D space, where project partners will share interest in commercial distribution company to be set up in Shanghai. Other setting up one or more companies in Shanghai, we will also build:
- Goldsmiths Shanghai Research Centre
- Social VR and online digital platform for knowledge exchange and collaboration, incorporating immersive technology
- leading UK arts, cultural, immersive and digital content partners
- leading Shanghai arts, tech, immersive and academic partners
The project will result in sustainable model boosting economic impact of immersive and strong legacy collaborative R&D space, where project partners will share interest in commercial distribution company to be set up in Shanghai. Other setting up one or more companies in Shanghai, we will also build:
- Goldsmiths Shanghai Research Centre
- Social VR and online digital platform for knowledge exchange and collaboration, incorporating immersive technology
Planned Impact
This project is built around creating direct impact for UK and Chinese creative industries. The primary beneficiaries will be the substantial list of project partners who participate in this project and their networks of collaborators. These will include the partners who have contributed letters of support to this project, but also further partners that we anticipate will take part in the project activities.
These creative industry partners will benefit through improved cross-national collaboration. In particular they will gain:
- A better understanding of the other nation's creative industry and market. This will be through presentations and networking at events.
- A greater understanding of how to do business and collaborate with the other country, also through presentation and networking
- Access to new collaborators in the other country. This will happen directly through networking opportunities at our events, in particular there will be frequent break-out group work that will facilitate working relationships
- Access to investment from both UK and China. This will be achieved through investor involvement in events and specific investor showcases.
- Access to markets in the other country. This will be through collaborations developed during the project, investment gained and better understanding of the other country. We will also set up a linked commercial vehicle to enable commercialisation of IP across nations.
These impacts can be measured in terms of financial value of investments (our minimum target is £10 million), number of collaborative ventures formed and the audience size for the creative outputs of those ventures.
While the immediate benefits will be to the participants, we do foresee a number of broader benefits.
The creative sectors of both countries will benefit more broadly. We aim to achieve a large enough scale of collaboration that its impact will spread to other companies. This will be due to creating a sufficient critical mass of individuals who have experience of cross-national collaboration and a more widely diffused understanding of cross-cultural working in both sectors. We expect the knowledge and collaborative relationships to spread through existing networks within the creative sectors of both UK and Shanghai (we have a number of partners with very strong networks). The project also includes the creation of durable institutions that will support further collaboration beyond the length of the project, including linked commercial vehicles in Shanghai and London to facilitate cooperation and IP exploitation and a joint research centre.
All of these impacts will have a positive economic impact on the UK creative sector and economy more broadly and strengthen the development of the Shanghai creative sector.
There will also be a positive impact on audiences in both countries through the creation of new creative work, and the greater creative innovation that comes from the cross-fertilization of ideas across cultures.
These creative industry partners will benefit through improved cross-national collaboration. In particular they will gain:
- A better understanding of the other nation's creative industry and market. This will be through presentations and networking at events.
- A greater understanding of how to do business and collaborate with the other country, also through presentation and networking
- Access to new collaborators in the other country. This will happen directly through networking opportunities at our events, in particular there will be frequent break-out group work that will facilitate working relationships
- Access to investment from both UK and China. This will be achieved through investor involvement in events and specific investor showcases.
- Access to markets in the other country. This will be through collaborations developed during the project, investment gained and better understanding of the other country. We will also set up a linked commercial vehicle to enable commercialisation of IP across nations.
These impacts can be measured in terms of financial value of investments (our minimum target is £10 million), number of collaborative ventures formed and the audience size for the creative outputs of those ventures.
While the immediate benefits will be to the participants, we do foresee a number of broader benefits.
The creative sectors of both countries will benefit more broadly. We aim to achieve a large enough scale of collaboration that its impact will spread to other companies. This will be due to creating a sufficient critical mass of individuals who have experience of cross-national collaboration and a more widely diffused understanding of cross-cultural working in both sectors. We expect the knowledge and collaborative relationships to spread through existing networks within the creative sectors of both UK and Shanghai (we have a number of partners with very strong networks). The project also includes the creation of durable institutions that will support further collaboration beyond the length of the project, including linked commercial vehicles in Shanghai and London to facilitate cooperation and IP exploitation and a joint research centre.
All of these impacts will have a positive economic impact on the UK creative sector and economy more broadly and strengthen the development of the Shanghai creative sector.
There will also be a positive impact on audiences in both countries through the creation of new creative work, and the greater creative innovation that comes from the cross-fertilization of ideas across cultures.
Organisations
- Goldsmiths University of London (Lead Research Organisation)
- Shanghai Jiao Tong University (Collaboration, Project Partner)
- China National Academy of Arts (Collaboration)
- Cybernaut Investment Group (Project Partner)
- Marshmallow Laser Feast (Project Partner)
- Manchester International Festival (Project Partner)
- Shanghai Mahua FunAge Culture Media Co. (Project Partner)
- HTC VIVE (Project Partner)
- Mirror Pictures (Project Partner)
- Royal Shakespeare Company (United Kingdom) (Project Partner)
- ShanghaiTech University (Project Partner)
- Shanghai Theatre Academy (Project Partner)
- i2 Media Research (United Kingdom) (Project Partner)
- Raindance Film Festival (Project Partner)
- Shanghai Lilacs (Project Partner)
- Digital Catapult (Project Partner)
- FXG (Project Partner)
- Philharmonia Orchestra (Project Partner)
- Dream Reality Interactive (Project Partner)
- Nesta (Project Partner)
- Hammerhead (Project Partner)
- Fudan University (Project Partner)
Publications
Dobre G
(2022)
More than buttons on controllers
Dobre G
(2022)
Immersive machine learning for social attitude detection in virtual reality narrative games
in Virtual Reality
Gibbs J
(2022)
A comparison of the effects of haptic and visual feedback on presence in virtual reality
in International Journal of Human-Computer Studies
Hilton C
(2023)
Simulating Location-Based Experiences in VR
Hiton C
(2023)
Prototyping Large Scale Projection Based Experiences in VR
Description | We have published our research in developing methods for using AI to drive non-player character in virtual reality games. We found that we could detect players level of engagement in a social interaction with a game character. This is useful for game designers for their future game design - for instance, instead of shooting someone to win the game in traditional games, in future VR you could gain someone's trust using your body language. This work was a collaboration with the UK creative industry, and demonstrates UK's competitive edge in immersive technology. We have also been collaborating with Microsoft Research Cambridge, conducting user studies in the area of remote meetings. |
Exploitation Route | We have published research papers and preparing others for our findings. |
Sectors | Digital/Communication/Information Technologies (including Software) |
Description | Research and teaching collaboration |
Organisation | China National Academy of Arts |
Country | China |
Sector | Academic/University |
PI Contribution | Our project supported the administrative side of this collaboration. |
Collaborator Contribution | Professor Jonny Freeman's time was paid by CAA at a negotiated preferential rate. |
Impact | We collaborated with CAA for a joint course focusing on creative topics. A first pilot online course was conducted on Human-Computer Behaviour and Psychology at CAA, run by both their lead Lecturer and one of our project co-Is (Prof Jonny Freeman). The course included a group presentation with various topics such as unconscious design-affordance and behaviour, diversified representation of image symbols. Students participated to showcase their work as the outcome of the course. The goal of this collaboration is to provide students with a comprehensive understanding of the various creative topics through a combination of instruction and hands-on experience. |
Start Year | 2022 |
Description | Research and teaching collaboration |
Organisation | Shanghai Jiao Tong University |
Country | China |
Sector | Academic/University |
PI Contribution | The AHRC seeding fund was used to organise a trip to Shanghai, which led to multiple meetings with Shanghai Jiaotong University. |
Collaborator Contribution | They hosted our meeting in China, and supported our further AHRC grant application. SJTU has offered a visiting Professorship to Prof William Latham from Goldsmiths. Dr Pearl Wang from SJTU has collaborated with us on market research, studying immersive content consumption by Chinese audience. |
Impact | We have signed an MoU with Shanghai Jiaotong University as a direct outcome of the AHRC seeding grant. |
Start Year | 2019 |
Description | AWE Asia Online Talk |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Industry/Business |
Results and Impact | PI Sylvia Pan and Co-I Marco Gillies gave an invited online talk at AWE Asia, a well-attended conference in China. |
Year(s) Of Engagement Activity | 2020 |
URL | https://www.awexr.com/asia-2020 |
Description | Beyond Conference - Creative China presentation and session MC |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Industry/Business |
Results and Impact | For the first time BEYOND, with the support of the AHRC's UK-China Creative Partnerships Programme, introduced Creative China, an exciting new track that explored research and industry partnerships between the creative industries in the UK and China. Creative China featured conference sessions including live Q&A plus a showcase of projects collaboratively developed with UK and Chinese creative organisations. Delegates discovered Shaun the Sheep in Shanghai, how to Create Shared Experiences for Multicultural Audiences with speakers from the Science Museum, Aardman, the Shanghai Theatre Company, HTC amongst many others. The IIIE project not only presented our research, the PI (Sylvia Pan) was the MC of the Creative China session. We estimate ~100 audience attended the Creative China session, and the video recordings from the session would reach to hundreds more. |
Year(s) Of Engagement Activity | 2020 |
URL | https://beyondconference.org/creative-china |
Description | Frontiers Panel Discussion - XR for bringing people together in times of lockdown and post-COVID |
Form Of Engagement Activity | A formal working group, expert panel or dialogue |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | I hosted a panel discussion as part of Frontiers in VR online talk series. One of them was particular related to China as we have invited the CEO of NReal glasses to join us discussing remote working. The video itself has been watched over 1,600 times as of March 2021 |
Year(s) Of Engagement Activity | 2020 |
URL | https://www.youtube.com/watch?v=nJ6ylZLwPkE |
Description | Immersive Assembly |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Held between October - December 2021, Immersive Assembly is a multidisciplinary, virtual residency bringing together an international cohort of artists and technologists Immersive Assembly is a programme for artists and technologists to learn, share, network and play. The cohort consists of 9 emerging practitioners from Argentina, China, Ghana, Mexico, United Kingdom and Zimbabwe, curated in partnership with our project partners, British Council and Goldsmiths, University of London. The 3 month residency programme supported the artists in developing new ideas for immersive experiences that invite us to collectively share in rituals and connections. This residency programme includes a series of sessions hosted by leading UK digital artists in Mediale's community, including art collective Marshmallow Laser Feast, artist and film-maker Kit Monkman, and Alexa Pollmann, director of performance design collective Peut-Porter. These sessions will inspire the group to further explore the theme of rituals and connection before arriving at their final idea, which they will pitch to a panel of experts from across the immersive arts industry in December. |
Year(s) Of Engagement Activity | 2021 |
URL | https://yorkmediale.com/immersive-assembly/ |
Description | Interview for national news - BBC Radio 3 |
Form Of Engagement Activity | A press release, press conference or response to a media enquiry/interview |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Public/other audiences |
Results and Impact | I was interviewed at the BBC Radio 3, talking about using VR and AI to make technology more inclusive, also about the future of UK-China partnership. It was a 45 minutes interview recently podcasted, already resulting in emails asking me to author a book in this area and some keynote speak invitations. |
Year(s) Of Engagement Activity | 2020 |
URL | https://www.bbc.co.uk/programmes/p085mn2k |
Description | Tedx talk: Social Interaction in the Era of Virtual Reality (also a version with Chinese subtitle) |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | The PI of the IIIE project, Sylvia Pan, has given a TEDx talk about social interaction in the time of virtual reality, where she promotes the value of research in VR, and her view of how VR is going to change our real lives. The talk received nearly 1000 views on YouTube. Subsequently, supported by contacts developed as part of the IIIE project, we were able to produce a Chinese subtitled version of this talk and distributed on various Chinese social platforms. It has been viewed over 10,000 times in China. As a result, we have received very positive comments and invitations to collaborate. |
Year(s) Of Engagement Activity | 2021,2022 |
URL | https://www.youtube.com/watch?v=rh6e-mW_y5E |