4i: Immersive Interaction design for Indie developers with Interactive machine learning

This research comes at an important time in the development of virtual reality and immersive media, with a low cost, mass market devices (the Oculus Rift and HTC VIVE) being launched for the first time. The launch of the devices is being accompanied by the release of associated motion controllers, such as the VIVE controllers or Oculus Touch. The UK games industry is world leading and an important sector of the economy, with a total market value of £4.19bn in 2015 and is the largest games development industry in europe, with over 1900 video games companies. Virtual Reality is the fastest growing sector of the UK entertainment industry and is predicted to reach £1.2bn by 2022, and be the largest in Europe (all statistics from UKIE, the UK Interactive Entertainment trade body, http://ukie.org.uk/). This expansion comes at a time in which small independent developers are increasingly important in the games and media industry. These small developers have increased the diversity and creativity of the industry, particularly with more work that is influenced by fine arts and literature (though there is considerable work still be done as women and BAME people are still underrepresented in the industry. As immersive media develop it is vital that independent developers and artists are able to play an important part to ensure that the medium fulfills its potential. This means not only small developers typical of the current games industry, but also populations that are currently underrepresented.

This project aims to enable independent developers and artists to design and implement movement based interaction for immersive media such as Virtual, Augmented and Mixed reality. The design and implementation process will be immersive in the sense that designing will occur inside the immersive medium, rather than at a desktop computer or on paper, and designing and implementation will happen by moving, so that designers can have a true sense of this movement. The key to this approach is Interactive Machine Learning (IML), where design is specified through examples of movement which are used as input to a Machine Learning algorithm, which "learns" to recognise those movements. However, this will be interactive: users will not simply gather a data and send it to the algorithm as a one off, but gradually add examples to refine and tweak the results, just as a design refines a product.

The tools will be developed collaboratively with users to ensure they meet the needs of our user groups and to understand how they perform immersive interaction design. This user research will be done, in the wild, with working developers and artists via a series of hackathons, game jams and choreographic coding labs. Since the challenges of this research are as much creative as they are technological, this method will be informed by arts practice in two ways. Firstly, we will work with developers to create interaction design workflows that centre around movement, these will be informed, and guided by the movement expertise of dance practitioners from the University of Coventry. Secondly, the short, user centred hackathons will be supplemented with a longer process of arts practice based research in which the tools will be used to create fully realised work, and the process of creation is reflected on. This practice research will be performed by Gibson/Martelli and two resident creators to be hired as part of the project.

The challenges of this project cannot be addressed simply with technology as the creation of immersive movement interaction is as much a creative as a technological problem. As such this project will be based on an close interaction between technology and artistic practice. The research will have 4 elements, the first two being technological and the second two artistic:
1) The development of immersive editing tools
2) User Centred Research
3) Movement Understanding from dance
4) Practice based arts research

The major beneficiaries of this project will be developers of Immersive Media (VR, AR and Mixed Reality experiences such as location based installations). They will benefit from having better tools and techniques for designing interaction, making it easier to develop more compelling forms of interaction that focus on users' body movements rather than on traditional button/joystick. This will not only enable them to make their experiences better but potentially to create new forms of experience, leading to new markets.

This project particularly focuses on small, independent developers, artists, and underrepresented groups within games and digital media. These SME developers are key both to the economics of the industry (over 95% of UK game developers are SMEs) and also to the cultural strength and diversity. Unlike major companies, they do not have access to the resources and specialist expertise current approaches to complex movement interaction and for the use of machine learning. It is therefore particularly important for them to have to types of usable and rapid tools that we intend to develop in this project.

This project will directly work with participants in these demographics in our hackathons. Our participants will be the first major beneficiaries, with immediate access to the software and training in its use. The software will be directly disseminated to developers as a plugin to a popular development platform such as Unity or Unreal Engine. This will make it easy for them to integrate it into their work. We will also develop a range of online training and support materials, including video, blog posts and MOOCs (see pathways to impact for details).

The result will be important economic impacts. The games industry is an important part of the UK economy (£4.19bn in 2015) and VR is the fastest growing sector of the UK entertainment industry (predicted £1.2bn by 2022). Supporting indie developers in creating new and innovative forms of immersive interaction in a cost effective way will support the general health of the immersive industry in the UK, and therefore economy in general.

There are also important cultural benefits. The tools we are developing will not only make it easier to create existing forms of interaction, but to create new forms of experience. This will help develop the cultural and artistic depth and diversity of immersive media, particularly if the tools are in the hands of creators out of the mainstream of games development, such as fine artists, women and other underrepresented groups (key participants in this project). This will help foster a rich and diverse culture within the new media of the 21st century, greatly benefiting the audiences for this media, opening them to new audiences and strengthening UK culture in general.

While the major impacts of this project are in immersive media, a secondary objective is to improve the usability of machine learning in general. ML is one of the most important technologies of the 21st century and has applications in a huge range of areas from medicine to advertising and finance to robotics. The impacts of machine learning are potentially huge in many sectors, however, it is still considered an obscure and difficult to use technology. Making it more usable will increase this impact by making the technology accessible to more people and sectors. Machine Learning could be used for tasks that it would not be used for otherwise because people working in those areas do not have machine learning expertise. Importantly, it may also help mitigate some of the negatives of the technology by expanding the scope of who can use it, therefore democratising machine learning. These impacts will be more indirect since the tools we develop will be specific to immersive media, however, we will disseminate general ML results through academic publications for researchers and experts and blog posts for broader public of ML practitioners and software developers.