Living with Digital Ubiquity

Lead Research Organisation: University of Nottingham
Department Name: School of Computer Science

Abstract

Since its formation in 2000 the MRL has focused on creatively interleaving the physical and the digital for everyday life. We have simultaneously explored new interactive possibilities and developed understandings of how these might be put to use in the real world. Much has changed over last 14 years; some of these changes we have driven, while others we have responded to.

When we began our journey, notions of ubiquitous, tangible and mixed reality computing were very much in their infancy, as were potential applications and methods. Our first seven years was therefore concerned with laying the conceptual and methodological foundations for these fields as well as demonstrating potential applications. Much of this was delivered through the Equator Interdisciplinary Research Collaboration (IRC) that we led until 2007.

We then used our previous platform grant to bridge the gap between Equator and the establishment of the Horizon Digital Economy Hub and Centre for Doctoral Training, and also the Orchid programme grant. The subsequent second stage of our journey has focused on driving the widespread adoption of ubiquitous computing as platforms such as smartphones and cloud infrastructures spread into everyday life. Now, as both Horizon Hub funding and Orchid imminently draw to a close in 2015, we need to once again reinvent our research agenda and also ensure our sustainability by bridging any funding gaps as we refresh our grant portfolio.

The focus of this next phase will be the overarching challenge living with digital ubiquity? The interleaving of physical and digital that was a distant vision back in 2000, is now becoming a reality for many, with a plethora of devices, from smartphones, to Kinects, to Fitbits allowing us to experience the digital in a bewildering variety of ways. The digital world has become rich, available 24/7, increasingly aware of our physical activities, and populated with our personal data. Ubiquitous technologies that were once considered radical are now becoming an unremarkable feature of our lives. Meanwhile, remarkable new technologies continue to emerge. Technologies for rapidly fabricating physical-digital products or for sensing and actuating the human body raise possibilities for radical new experiences.

Our vision of living with digital ubiquity is therefore one of balancing the challenges of living with increasingly mundane ubiquitous technologies, while also enabling people to experience and create extraordinary new experiences. We see this as a productive tension. People must both live in the everyday and yet also seek the profound and creative. It also reflects the essential - and highly creative - tension in the MRL between addressing everyday technology challenges as revealed by ethnographers, while simultaneously engaging with artists and performers to create provocative new experiences.

We therefore propose to exploit platform funding to catalyse a step-change in our research by identifying the new research challenges that will arise from living with digital ubiquity, exploring potential approaches to these challenges, developing our key researchers so that they are ready to face them, and forming the partnerships that will help us address them through future research projects. Platform funding will enable us to deliver a programme of exploratory projects, visitor exchanges, and impact projects to explore our vision while also enabling us to retain and develop key staff over the coming critical period.

Planned Impact

The MRL has a long history of delivering cultural impact by working with artists to create, tour and study interactive performances and installations that combine the physical and digital in unusual ways. Works produced since 2008 have toured to over 40 venues in 18 countries, have been experienced by over 200,000 people, and have received multiple arts awards. Cultural impact then drives economic impact, where we collaborate with major players in more mainstream entertainment to create new kinds of games and television shows. Prototype games produced with Sony and Nokia have garnered further industry awards, while our technologies and know-how have also been employed to produce articles for TV shows, from Blue Peter to Bang Goes The Theory and the One Show. In turn, these drive wider societal impact as our artworks often foreground and provoke public debate around societal challenges (e.g., works produced with the artists Blast Theory have tackled surveillance and terrorism and with Active Ingredient have tackled climate change), while features on mainstream TV shows draw the attention of the wider public to the potential of emerging technologies (e.g., highlighting the capabilities of wearable biosensors).

Platform funding will help us sustain this kind of impact in the future. Moreover, we will utilize the flexibility of a platform grant to pilot novel impact mechanisms that establish pathways to the new communities associated with our two research challenges. One important new pathway to establish concerns impact on the many millions of amateur and everyday creative practitioners that are increasingly engaging with digital technologies, and whose activities make a vital - if sometimes unsung - contribution to the economic and social wellbeing of the UK. A second, is to establish new pathways to impact on policy makers and shapers, including national and local government as well as campaign organisations.

Our proposal therefore includes a range of impact mechanisms, both to sustain our traditional pathways and also to open up these new ones. These include bespoke impact projects such as KE Secondments (supplemented with PhD internships) targeted at industry and policy partners; community engagements that take place in maker- and hack-spaces and other local venues where amateur and DIY makers can be found; and the release of products as impact probes, including downloadable apps, but also physical artefacts distributed through pop-up shops and craft fairs. Wider public impact will be enhanced by the production of a series of videos for the increasingly popular Computerphile YouTube channel. These mechanisms will be explicitly badged as 'impact' mechanisms so as to focus our researcher's attention on the increasing importance of impact.

Publications

10 25 50
publication icon
Ramchurn R (2018) #Scanners 2 - The MOMENT

publication icon
Kwon H (2017) "It's Not Yet A Gift"

publication icon
Greenhalgh C (2016) ^muzicode$

publication icon
Tennent P (2019) Abstract Machines

publication icon
Benford S (2016) Accountable Artefacts

publication icon
Chamberlain A (2016) Audio in place

publication icon
Darzentas D (2019) Card Mapper

publication icon
Avila J (2019) Encumbered Interaction

publication icon
Hazzard A (2019) Failing with Style

publication icon
Rennick-Egglestone S (2016) Families and Mobile Devices in Museums Designing for Integrated Experiences in Journal on Computing and Cultural Heritage

publication icon
Kwon H (2015) FugaciousFilm

publication icon
Greenhalgh C (2016) GeoTracks

publication icon
Rennick-Egglestone S (2016) Health Technologies 'In the Wild'

publication icon
Reeves S (2015) I'd Hide You

publication icon
Brown R (2016) IMorphia

publication icon
Benford S (2016) On Lions, Impala, and Bigraphs Modelling Interactions in Physical/Virtual Spaces in ACM Transactions on Computer-Human Interaction

publication icon
Luger E (2015) Playing the Legal Card

publication icon
Jäger N (2017) Reciprocal Control in Adaptive Environments in Interacting with Computers

publication icon
Malizia A (2016) Screwbots in Ubiquity

publication icon
Marshall J (2017) Touchomatic

publication icon
Porcheron M (2016) Using Mobile Phones in Pub Talk

 
Title Apocalypse of MoP 
Description As well as giving gamers the chance to enter an online world as a double agent and infiltrate a corrupt government, new online alternate-reality game Apocalypse of MoP also uncovers players' perceptions of provenance and how this affects their decision-making. The game involves players answering questions and completing tasks relating to the digital provenance of objects. These decisions will then be used to assess how people process information and how important provenance - the origin of information - is in decision-making. It was developed by researchers at The University of Nottingham's Mixed Reality Lab in collaboration with Nottingham-based production company Urban Angel and researchers at the University of Southampton. 
Type Of Art Artwork 
Year Produced 2014 
Impact This online game was played by over 900 players over the course of s year who completed 13,000 provenance tasks. The game was the focus of a full technical paper at the ACM's Computer Human Interaction (CHI 2015) conference. 
URL http://apocalypseofmop.com
 
Title Climb 
Description Ascore for an interactive -game-like classical piano work, implemented using our Muzicodes interactive music technology. 
Type Of Art Composition/Score 
Year Produced 2017 
Impact Toured to UK (Nottingham and London) an Finland. URL below points of archive of all performances. 
URL http://music-mrl.nott.ac.uk/1/archive/explore/Climb
 
Title Eye as Witness 
Description A touring exhibition that uses VR to engage the public with photographs from the holocaust, and especially with understanding who took the photos (often perpetrators0 and why, 
Type Of Art Artistic/Creative Exhibition 
Year Produced 2020 
Impact Initial tour 5/1/20 - 27/3/20 Imperial War Museum North, Salford JANUARY - FEBRUARY 2021, Nottingham Lakeside Arts Centre 9/1/20 - 29/3/20, The Peace Museum 27/1/20 - 30/1/20, South Hampstead Synagogue 8/4/20 - 28/6/20 The National Memorial Arboretum 
URL https://witness.holocaust.org.uk/venues-and-dates
 
Title Halfway to the Future exhibition 
Description A retrospective exhibition of the Mixed Reality Lab's creative work over the past 20 years 
Type Of Art Artistic/Creative Exhibition 
Year Produced 2019 
Impact Staged as the centre piece of the CHI 2019 conference exhibition when it cam to Glasgow in 2009. Attended by up to 3,500 delegates. 
 
Title Karen 
Description Created by the artists group Blast Theory, Karen is an interactive artwork that explores the topic of psychological profiling by engaging audiences with an interactive video narrative delivered by a mobile app. We provided technical support fro delivering Karen and also supported a visiting residency for Nick Tandavanitj from Blast Theory. 
Type Of Art Artwork 
Year Produced 2014 
Impact Presentations: Film Festival, Storyscapes at Tribeca Film Festival, New York, 2015 Exhibition, Stedelijk Museum Bureau Amsterdam, 2015 Exhibition, ZKM Center for Art and Media, Karlsruhe, 2015 Film Festival, Festival du nouveau cinema, Montreal, 2015 Geneva International Film Festival, Monokino, Geneva, 2015 Awards: Data Category, Best of British Digital, British Interactive Media Association, Winner, 2015 Innovation Award, Festival du nouveau cinéma, Winner, 2015 The People's Lovie Awards, Experimental & Innovation, The Lovie Awards, Bronze Winner, 2015 
URL http://www.blasttheory.co.uk/projects/karen/
 
Title Oscillations 
Description Virtual reality swing rise that played with sensory discrepancies between visual and kinaesthetic stimuli. 
Type Of Art Artwork 
Year Produced 2016 
Impact Exhibition at Sheffield DocFest 
 
Title Scanners 
Description Scanners is a new immersive art installation that literally reads your brain waves and allows you use the power of your thoughts to edit the film you are watching. Created in partnership with arts group Albino Mosquito, Scanners is an interactive film controlled by a brain-computer interfaces. While the film always runs for 17 minutes and follows the same plot, viewers' blinking triggers cuts between views of reality and dreams while their levels of attention and meditation control the mixing of four different video layers. 
Type Of Art Film/Video/Animation 
Year Produced 2015 
Impact Scanners premiered at The Foundation for Art and Creative Technology (FACT) in Liverpool in July 2015 which it was screened for a week. It has also been shown at the ACM Creativity and Cognition (C&C 2015) and Computer Human Interaction (CHI 2016) conferences. 
URL http://www.albinomosquito.com/2013/06/scanners-teaser/
 
Title Sentiment 
Description An interactive installation in which participants navigate an array of speakers to piece together interviews with people about traumatic events and where the galvanic skin respond of the those people captured during the interviews is replayed as a series of vibrations on the listener's body. 
Type Of Art Performance (Music, Dance, Drama, etc) 
Year Produced 2015 
Impact Staging of the exhibition for the Foundation for Art and Creative technology (FACT) at Liverpool. 
 
Title The Body 
Description A theatrical performance incorporating sensors and output devices embedded into physical props (dolls) and seating so that the audience can interact with the performance. The work was created in collaboration with the theatre group Shunt, supported by a prestigious award from the Samuel Beckett Trust and premiered at London's Barbican Theatre in November 2015. 
Type Of Art Artwork 
Year Produced 2015 
Impact It received excellent reviews including four star reviews from both the Guardian and The Stage. "Arguably more installation than theatre, The Body is bold in its rejection of story and embrace of technology. Those hoping for plot or character will be disappointed, but as a set of images and sensations it's often breathtakingly beautiful." (The Guardian, ****) http://www.theguardian.com/stage/2015/nov/22/the-body-review-barbican "Work of this calibre is worth shouting about" (The Stage, ****) https://www.thestage.co.uk/reviews/2015/the-body-review-at-the-pit-barbican-centre-london-bold-and-off-kilter/ "Impressive work by the University of Nottingham's Mixed Reality Lab" (LONDONist, ***) https://londonist.com/2015/11/hello-dollies-the-body-barbican-pit 
URL https://www.thestage.co.uk/reviews/2015/the-body-review-at-the-pit-barbican-centre-london-bold-and-o...
 
Title The Moment 
Description A brain controlled movie using EEG to create an individual individual through each screening. 
Type Of Art Film/Video/Animation 
Year Produced 2018 
Impact Touring to film and other festivals as well as screenings at scientific conferences, including: Nottingham Festival of Science and Curiosity 15th February 2020, Sneinton Market, Nottingham UK Wonder, University of Nottingham 15th June 2019, Nottingham UK Spark Festival British Council in Hong Kong 18th, 19th 20th January 2019 Brain Film Festival, Barcelona 15th - 17th March 2019. Barcelona, Spain Brain Film Festival, Madrid 19th March 2019, Madrid, Spain River College Virtual Reality Workshop 22nd March 2019, Padua, Italy LEVEL, in collaboration with East Midlands Participatory Arts Forum (EMPAF) 11th April 2019, Derby CHI 2019, SECC Glasgow 6th - 9th May 2019 Glasgow The Creative and Digital Priority Area Broadway, Nottingham 17th May 2019 Pint of Science - Talk by Richard Ramchurn Parliament Bar and Square, Nottingham 21st May 2019 Pint of Science - Screenings of The MOMENT Canal House, Nottingham 23rd May 2019 Wonder, University of Nottingham 15th June 2019. Touring 2018 World Premier at Sheffield Doc/Fest 2018 7th - 12th June 2018. Howard Street, Sheffield, UK BlueDot Festival 20th and 21st July 2018. Jodrell Bank Conservatory, Cheshire, UK Kendal Calling Festival 26th - 29th July 2018. Lowther Deer park, Lake District, UK Ars Electronica 6th - 10th Spetember 2018. Linz, Austria Arts By the Sea 29th and 30th September 2018. Bournemouth, UK Continue Conference 2nd October 2018. York, UK Reykjavik International Film Festival - Live Score Screening 6th October 2018. Reykjavik, Iceland. Mayhem Film Festival, Broadway, Nottingham 11th - 14 October 2018 Leeds International Film Festival, Caravan Screening 1st - 3rd November 2018 Geneva International Film Festival - Auditorium screening and roundtable 5th November 2018, Geneva, Switzerland Aesthetica Short Film Festival 7th - 11th November 2018. York, UK Lakeside Arts -Djanogly Recital Hall - Live Score Screening 16th November 2018. Nottingham. UK 
URL http://braincontrolledmovie.co.uk/tour/
 
Title The Moment Live Score 
Description Live score musical performances of the score from the Brain-Computer controlled film The Moment. Live musicians play the score while video projections are created using the BCI interface. 
Type Of Art Performance (Music, Dance, Drama, etc) 
Year Produced 2019 
Impact Touring programme from 2020 onwards. 
URL http://braincontrolledmovie.co.uk/tour/
 
Title Thresholds 
Description Restating the world's first photographic exhibition in virtual reality 
Type Of Art Artistic/Creative Exhibition 
Year Produced 2017 
Impact Yes to arise 
 
Title Touch-o-matic 
Description A two player arcade game that measures how much players are touching one another as a basis for controlling the game. 
Type Of Art Artistic/Creative Exhibition 
Year Produced 2015 
Impact Experienced by 2000 members of the public when exhibited at the National Video Game Archive in Nottingham. 
 
Title VR Playground 
Description Series of interactive rides that play with sensory discrepancies between visual and kinaesthetic stimuli in virtual reality 
Type Of Art Artwork 
Year Produced 2017 
Impact Planning a major touring programme beginning with the Norwich and Norfolk Festival in May 2017 
 
Description A variety of interactive techniques for integrating ubiquitous technologies into everyday life and demonstrations of future applications.
Exploitation Route Used by various commercial partners including BBC, B3 Media and Unilever.
Sectors Creative Economy,Culture, Heritage, Museums and Collections,Retail

 
Description The overall goal of this platform grant was to catalyse a step-change in research at the Mixed Reality Laboratory and so ensure its future sustainability. This mapped onto three strategic objectives: (i) charting new research challenges that arise from living with digital ubiquity; (ii) fostering new partnerships to enable us to explore these challenges, including industry, artistic and policy partnerships as well as academic partnerships; and (iii) extending the future impact of our research by establishing impact pathways. The flexibility of platform funding proved to be a powerful catalyst for opening up new avenues of research, with the grant directly supporting the production of many papers. Conference papers included 15 papers at the ACM CHI conference, 2 papers in ACM CSCW, and also papers in SIGIR, DIS and NIME. Journal papers include ACM Transactions on CHI, Journal of Computing and Cultural Heritage, Information Processing and Management, Personal and Ubiquitous Computing, and Interacting with Computers. Many of these papers opened up new research avenues, involved external academic, artistic and other industry collaborations, and were associated with significant additional impacts. Key highlights are: - Development of brain-computer interfaces, both for assessing usability in work settings but also for artistic applications through a collaboration with artist Richard Ramchurn to produce, tour and two brain-controlled films leading to new conceptual insights into how humans contest control with digital systems and their own bodily responses to these. - Establishing a new research agenda around sensory misalignment in immersive experiences through a collaboration with artist Brendan Walker to create and tour a series of immersive rides. - Opening up questions about how performers interact with autonomous musical instruments, how audiences understand this, and the aesthetics of failure and improvisation through a collaboration with composer and pianist Maria Kallionpaa to create and tour her interactive piano work Climb! - Exploring the potential of interactive mirrors in performances and developing the concept of performative mirror space through a collaboration with artist Rachel Jacobs. - Creating multiple decks of ideation cards and a supporting software tool for collecting data about patterns of their use in areas such as moral IT and mixed reality game design. - Creative explorations of digitally augmenting guitars led to a new concept of 'encumbered interaction' and from this to a new international collaboration with the Royal Institute of Technology on the somaesthetic design of embodied interaction with digital technologies. - A new collaboration with formal computing researchers at Glasgow to explore how the formalism of bigraphs (originally established by Robin Milner) can be more widely applied to the practical design and understanding of ubiquitous computing systems. - A new collaboration with The University of Illinois to explore human search strategies on the Internet. - A thread of work exploring how to wrap physical products in layers of digital experience, from guitars that capture and tell stories of who is playing them to digital gift wrap for food, the latter leading to a major ongoing collaboration with Unilever to explore digitally augmented consumer goods. Not only is each of these examples reflected in the publication of research papers, but also in impacts and public engagements, especially though touring artworks that have reached tens of thousands of direct participants as they have toured to cultural venues and festivals worldwide. Ultimately, the success of this platform funding in refreshing our research agenda and building partnerships has led to the Mixed Reality Lab's involvement in several major new initiatives that have secured our longer-term sustainability. - The Horizon Digital Economy Research Hub and associated Centre for Doctoral Training, led by Koleva and Benford respectively, were both refreshed in 2019-2020 largely based on a vision and partnerships that had emerged from our platform grant. - We are partners in the UKRI National Hub for Trustworthy Autonomous Systems and also in the associated Governance node, again building on explorations of human interaction with autonomous systems carried out in the platform grant. - Our University has since invested several million pounds into establishing the Smart Products Beacon of research excellence, led by Benford, to forge new research collaborations between Computing, Engineering and other disciplines, including establishing a new Co-bot Maker space. - We have recently been awarded ERDF funding to deepen future impact by supporting creative digital SMES in our region. Finally, 2019, the end of the platform grant, marked the 20th anniversary of the Mixed reality Laboratory and so we used platform funding to support two further dissemination and impact activities. We hosted the first Halfway to the Future conference in Nottingham, attended by 70 international delegates. We staged a major retrospective exhibition of our interactive artworks as the centrepiece of the ACM CHI 2019 conference in Glasgow that was attended by over 3,000 delegates.
First Year Of Impact 2015
Sector Creative Economy,Government, Democracy and Justice,Culture, Heritage, Museums and Collections,Retail
Impact Types Cultural,Societal,Economic,Policy & public services

 
Description ADAM: Anthropomorphic Design for Advanced Manufacture
Amount £269,486 (GBP)
Funding ID EP/N010280/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 01/2016 
End 07/2017
 
Description Digitally Enhanced FMCG
Amount £426,000 (GBP)
Organisation Unilever 
Department Unilever Research and Development
Sector Private
Country United Kingdom
Start 11/2015 
End 10/2018
 
Description FAST Programme Grant
Amount £5,119,944 (GBP)
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 06/2014 
End 06/2019
 
Description GIFT - European Union Horizon 2020
Amount € 2,000,000 (EUR)
Organisation European Union 
Sector Public
Country European Union (EU)
Start 01/2017 
End 12/2020
 
Description Horizon Follow-On Fund
Amount £4,062,954 (GBP)
Funding ID EP/M02315X/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 08/2015 
End 07/2020
 
Description B3 Media 
Organisation B3 Media
Country United Kingdom 
Sector Charity/Non Profit 
PI Contribution We have jointly run three iterations of the annual the Talentlab Programme to recruit and train young black and minority and ethnic (BAME) artists in the design and creation of interactive artworks.
Collaborator Contribution B3 media have coordinated the programme, providing core training, recruiting artists and working to secure opportunities for presenting the work to sponsors in the creative industries, for example a showcase at BAFTA.
Impact To date we have trained 30 new artists by running workshops, hosting residencies and supporting the development of prototypes.
Start Year 2014
 
Description Chinese University of Hong Kong 
Organisation Chinese University of Hong Kong
Country Hong Kong 
Sector Academic/University 
PI Contribution Supporting Christian Greiffenhagen as a visiting researcher to our lab.
Collaborator Contribution Collaboration on a series of research studies.
Impact Papers under preparation
Start Year 2016
 
Description FACT 
Organisation Foundation for Art and Creative Technology (FACT)
Country United Kingdom 
Sector Charity/Non Profit 
PI Contribution We developed a exhibition called Performing Data for them.
Collaborator Contribution They hosted the exhibition, delivering an estimated audience on excess of 1000 visitors. They subsequently have engaged as external supervisors for a PhD student.
Impact In prepearation
Start Year 2015
 
Description Matt Collishaw 
Organisation Mat Collishaw
Sector Private 
PI Contribution Helming develop a virtual reality artwork based on restaging the world's first photographic exhibition
Collaborator Contribution Development of said artwork and securing opportunities to display it in public and supporting pruning from Arts Council.
Impact Artwork in preparation and papers to follow.
Start Year 2016
 
Description Unilever 
Organisation Unilever
Department Unilever UK R&D Centre Port Sunlight
Country United Kingdom 
Sector Private 
PI Contribution Developing concerts, enabling technologies and prototypes to explore how augmented fact moving consumer goods connect to smart appliances to augment to consumer experience as key touchpoint during their interaction.
Collaborator Contribution Direct contributions to research from Unilever staff as well as cash funding for our research.
Impact Four patents filed. Papers in preparation.
Start Year 2015
 
Company Name MAKERS OF IMAGINARY WORLDS LTD 
Description Producing interactive installations and performances for children. 
Year Established 2019 
Impact 'The Musical Meadow' Tour MOIW have developed 'The Musical Meadow' a playful and fun interactive interface that appeals to children's needs and instinctual behaviours to interact, touch and exert their agency. The meadow can be experienced at events in the UK. Touring Dates: 28th June Curve Theatre Foyer, Leicester. The Mighty Creatives, Creating the Future. 'The Coral Garden' Set in a 4 meter interactive dome The Coral Garden connects children with exciting technology and responds to their touch. Traditionally crafted hyperbolic crochet corals hand made by local knit and knatter groups are interwoven with modern technology to create a truly engaging experience. Come and light up the Garden with you voice and see what sounds you can create tickling the urchins. Dates and Times: 12-14 February 12.30-6:30pm 15th February - 9-1pm Special Day for young people with Special Educational Needs and Disabilities. 17-19 February- 9-6:30pm Storytelling inside The Coral Garden: 12 - 14th February 10am and 12 noon. Please book your place for the performance at St Anns Library, details below. Venue: St Anns Library, St Anns Valley Centre, 2 Livingstone Road, Nottingham, NG3 3GG. Tel 0115 883 9700.
Website http://makersofimaginaryworlds.co.uk/