Next generation paper

Lead Research Organisation: University of Surrey
Department Name: Digital World Research Centre

Abstract

The technology of paper is so old and so ubiquitous that we have stopped thinking of it as a technology. Nevertheless, paper and ink has continued to evolve from the earliest experiments to record handwriting on parchment in China 2000 years ago, to the medieval printing press in Germany, and modern advances in the printing of true colour digital images. Until recently, these developments have treated paper as a passive reading surface for the presentation of visual information. However, the advent of printed electronics and augmented reality have brought a new interactive possibility to paper: that it could communicate with things around it. In particular, it can be made to trigger the presentation of overlaid or adjunct information in both visual and auditory form on nearby digital devices. This is a potentially revolutionary development of paper for the digital content economy, because it begins to tie together as never before, traditionally separate media forms of printed books and magazines with music, film and information on the world wide web. Paper can become both a carrier of information and an interface to further information beyond itself.

In this project, we aim to develop these properties into a generic mass media 'next generation' paper. We will harness developments in both image recognition and printed electronics to explore two technical trajectories for the support of interactive paper over the short and long term, within a common interaction framework. This framework will be tied into new and emerging MPEG standards for media linking, to allow media producers to author hybrid print-and-digital interactive documents in a common international format. The framework itself will be generated by a multidisciplinary team of scientists and engineers working in a promising domain of application, and embodied in an authoring toolkit for creating interactive e-books with printed components. Computer scientists and electronic engineers will collaborate with media and interaction designers, social scientists and business researchers to design, explore and test the paradigm of interactive printed books for travel and tourism.

The travel domain is both promising and challenging, because it still makes use of both printed and online information in professional content to guide tourist choices, and in the capture of tourist reviews and experiences in social and personal media. For this reason we will work with travel companies and the publishing industry to generate a series of professional travel books which have interactive properties for playing back additional content on a variety of devices. We will also work with tourists themselves to evaluate the usability and usefulness of these interactive travel books, and to create multimedia photobooks or travelogues of their own travel experiences, with content separated across paper, screen and speakers. These investigations will be informed by a prior ethnographic study of the use of paper and screen-based information across the tourist journey, to identify the problems and opportunities for combining them in new ways. They will also be underpinned by core technology development to realise a practical and robust way of triggering media playback from page turning and pointing actions over the surface of a printed book, sensed either by an overhead camera or by light sensitive patches wired to a wireless chip in the spine of the books. Finally, the commercial potential of the technology will be examined and defined in a dedicated business investigation in the second half of the project. This will involve consideration of the immediate and short term potential of image recognition for interactive paper, as well as the longer term potential of printed electronics where a camera is not required.

Planned Impact

A next generation paper (beyond '1GP') will open up countless opportunities, enabling new forms of graphic design for hybrid physical-digital documents in which content is shared between paper, speaker and screen. It will support new forms of multimedia literacy and expression that extend currently separate forms of expression in the creative industries stemming from film, music, radio, TV, books and the web. It will also make hidden digital content more tangible and accessible, allowing simple indexes to be printed in a variety of forms for circulation, display and storage in the physical world. Consequently, we envisage a very significant impact, in the following domains.
KNOWLEDGE: The project will impact all the academic communities represented by the project team (see also Academic beneficiaries). For example, in augmented paper and interaction design the project may realise new benefits of augmentation, and explore tangible paper interfaces to computing. In the computing area the project will extend previous architectures for cross-media representation and linking into the paper domain, and extend international ISO MPEG standards to include physical objects and interactions. In printed and embedded electronics, the project will advance the development of practical paper and flexible electronics, and showcase a new application to augmented books. In cross-media publishing, the project will enable transmedia storytelling with paper. In tourism and digital geographies, the project will extend augmented reality to paper and contribute new theory on the cultural concepts of 'convergence culture' and 'interface'. In new business innovation, it will contribute to new theories and approaches to agile innovation in the media industry, where platform, content and experience interact to produce new media experiences.
PEOPLE: The project may enable new career roles in the creative industries, for hybrid print-and-digital content creators and entrepreneurial thinking. It will also increase the complexity of publishing systems, and could lead to new forms of paper production, requiring new training in media publishing and paper manufacturing.
ECONOMIC: The findings of the project will be of immediate benefit to the tourist industry, which was worth £121 billion to the UK economy in 2014 (UK Tourism statistics 2016). However, the longer-term impact will be on the global print and e-book publishing industry, through a research and development contribution to its current debate about the future of paper in a digital age. Hence the project will accelerate the use of cross-media content, initially in the travel domain, by delivering: (a) technological solutions to bridge the content gap between the material and virtual worlds, (b) solutions to bring together user-generated content and industry generated content, (c) easy software solutions to create cross-media content involving interactive paper. We anticipate a two stage rollout of project technology in the digital economy, with 2GP paper based on paper-and-hand image recognition, and 3GP paper based on flexible printed electronics with wireless communications. At both stages the solution will leverage the simplicity, ubiquity and accessibility of paper with a user model that seamlessly activates adjunct content from simple reading gestures, and a cost point which is affordable to all.
SOCIETAL: Societal: The project will amplify societal trend towards transmedia reading across devices, and enable both professionals and prosumers to create connections across physical and virtual media, easily and cheaply. This will lead to a twofold impact on ordinary people as both consumers and producers of interactive paper, with benefits in the form of new reading and writing behaviours. The capability to augment and "virtually" update printed documents reduces the requirement for documents to be reprinted, thus reducing global paper consumption.

Publications

10 25 50
 
Title Augmented content for a-book 
Description Images, audio, video and videos of panning image slideshows with audio overlay were created and edited for inclusion in an augmented travel guide using visual recognition technology and touch sensitive sensors embedded in the paper. These were generated collaboratively by the travel writer and the designer of the guide. The travel writer recorded the material in-situ and the designer edited and prepared these for inclusion in the augmented guide. 
Type Of Art Artwork 
Year Produced 2019 
Impact A fully augmented travel guide book was created with the artwork generated, such as images, audio, video and videos of panning image slideshows with audio overlay, made accessible through a mobile app that uses visual recognition technology. This was released as a commercial product by Bradt Travel Guides on February 25th 2019 and is now freely available to the public. The travel book can be purchased for a fee and the app can be downloaded for free through the Google Play store. 
URL https://www.bradtguides.com/cornwall-the-isles-of-scilly-slow-travel.html
 
Description We are about half way through the project so findings are still emerging. Some preliminary findings are given after each objective below:

1. What is the relative role of paper and screen-based information in tourism?
Our ethnographic study showed active use of paper by all ages of travellers, in combination with online resources. Travellers require a central place to collate material as they continuously adjust their travel plans before and during a trip. We believe the augmented travel guide could function at this place and have adjusted its design to support personalisation and collection of digital assets.
2. What authoring platform is required to create interactive printed books that interwork with internet devices?
We have built an initial platform to demonstrate both 2G (optical) and 3G (embedded electronic) paper in an augmented guide to Cornwall. Our engagement with publishing partners was key to ensuring that it fits with the publishing workflow and uses industry-standard authoring such as Adobe InDesign to author augmented books (we are now calling 'a-books').
3. What form of image recognition and printed electronics solutions are needed to support interactive printed books that respond seamlessly to human touch?
We utilise a visual search engine from a subcontractor Visual Atoms, and this is performing well on 'page content' that it was never designed for. We have also implemented 'light tags' in a full 3G demonstrator book which uses light sensitivity to identify which page is open. We had to invent a new method of wiring tags from every page to an embedded chip in the binding, and have filed invention disclosures for this and alternative methods of page identification.
4. What is the value and experience of printed compared to screen-displayed books, as an interface to additional digital content on a second screen?
Lab and field-based studies are underway to examine this. Early findings suggest some personality preferences for either paper or screen, and an intriguing relationship between
e-books and a-books we do not yet understand. Some users want to use paper-and-screen in planning a trip and then take the e-book for reference.
5. What new business opportunities are enabled by a new class of interactive printed books based on image recognition and light sensitive paper?
Travel and tourism has turned out to be a very promising domain of application, since our commercial partners have already asked to publish a full augmented guide to Cornwall with our experimental Android app. So we are now discussing a full commercial app and licensing arrangement with project partners in the first instance. In addition we have been approached from other paper, publishing and advertising companies for access to our technology, and had strong interest in applications to augmenting textbooks for education. Our continuing business investigation is following up these leads with Travel Industry and Other Industry focus groups this year.
Exploitation Route We have good engagement with the electronic publishing industry and have already presented our 'a-book' approach at conference in that field. We also have an overview team publication in press for the Journal of Electronic Publishing; invited by the Editor-in-Chief. This is a primary audience for uptake of the approach which could be quite quickly adopted using our 2G solutions.

The travel industry is another sector in its own right for application and we are continuing to engage with them through our business investigation and new work on augmented photobooks with a photobook publisher. This will reveal how to support self-made a-books by consumers.
Sectors Creative Economy,Digital/Communication/Information Technologies (including Software),Education,Electronics,Environment,Healthcare,Leisure Activities, including Sports, Recreation and Tourism,Government, Democracy and Justice,Culture, Heritage, Museums and Collections,Retail

URL https://www.surrey.ac.uk/digital-world-research-centre/funded-projects/next-generation-paper-connecting-paper-web
 
Description We have already demonstrated the project technology at public exhibitions in The Hague and Amsterdam, as well as at several academic/industry conferences (see engagements). The project has also attracted a good deal of media attention resulting in about 16 articles on first announcement of the grant, to further articles throughout the first 16 months. The latest publicity relates to the release of the commercial travel guide to Cornwall by Bradt Travel Guides with our experimental app.
First Year Of Impact 2017
Sector Leisure Activities, including Sports, Recreation and Tourism
Impact Types Cultural,Economic

 
Description Bradt Travel Guides 
Organisation Bradt Travel Guides
PI Contribution We have added digital links to a travel guide to Cornwall and provided an app through which these can be played whilst reading the guide. This was released onto the market in March 2018: https://www.bradtguides.com/cornwall-the-a-book
Collaborator Contribution The partners have provided continuous advice to us on the publishing workflow for their travel books. This has influenced the design of our app but also our approach to make it more attractive to the publishing industry. In addition they have worked with us to annotate the Cornwall guide with icons indicating hotlinks to digital content and advertised the Surrey app by which these can be read by their customers.
Impact The Slow travel guide to Cornwall and the Isles of Scilly, with accompanying 'Cornwall a-book' app.
Start Year 2017
 
Title Cornwall a-book 
Description A team of researchers at Surrey and the Open University have been working with Bradt Travel Guides on an innovative Android app, Cornwall a-book, as part of the Next Generation Paper (NGP) project - a marriage between classic paper and digital technology. Cornwall a-book is an augmented book app that is working with the world's first augmented travel guidebook "Cornwall & The Isles of Scilly" that utilizes visual object recognition, audio analysis and geo-location to provide exciting, personalised digital content to the traveller. This breaks the boundaries between paper and digital content to transform the travel experience 
Type Of Technology Webtool/Application 
Year Produced 2019 
Impact Cornwall a-book is released at Google Play store. The app supports 8036 types of Android devices, and is made available to 144 countries world wide. Cornwall a-book uses the latest advances in AI to robustly and instantly recognise and classify real-world objects and locations, even under changing viewing angles and obstructions. This technology will make the travel experience more comfortable and engaging. In addition, the launch of this app is an exciting next step in shaping the future of books, paper and technology in the travel industry. The augmented guidebook embeds multimedia content within traditional guidebooks enabling travellers access to interactive information without having to put their book down. Furthermore, the release of Cornwall a-book with Bradt Travel Guides is an important milestone for Next Generation Paper. It provides a great use case for how this marriage between physical and digital media could greatly improve people's experience and knowledge of the world around them. 
URL https://play.google.com/store/apps/details?id=cvssp.uos.hy0006.ngp_cornwall
 
Description 'From postcards to Instagram: Re-imagining tourist landscapes' paper presented at RGS-IBG Annual International Conference, Cardiff University, Cardiff, 29 August 2018. 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact 'From postcards to Instagram: Re-imagining tourist landscapes' paper presented at academic audience, RGS-IBG Annual International Conference 2019, Cardiff University, Cardiff
Year(s) Of Engagement Activity 2018
URL http://conference.rgs.org/AC2018/98
 
Description 'Introducing the a-book'. Keynote presentation to BIC New Trends in Publishing conference 2018. 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Industry/Business
Results and Impact David Frohlich gave an invited presentation on 4th September 2018 to the Book Industry Communications conference on New trends in publishing. The talk was entitled 'Introducing the a-book' and presented the work of the project so far in defining a new category of print publication to be read with digital links shown on a nearby smartphone or other device. There was considerable interest from delegates in how this might change publishing processes and reader behaviour. There was an interesting connection between our talk and others describing the decline of e-book sales and the rise of audio book sales, suggesting an opportunity for audio-enhanced printed books.
Year(s) Of Engagement Activity 2018
URL http://www.bic.org.uk/198/New-Trends-2018/
 
Description 'Journeys Between Print & Screen and Beyond: A Travel Media Ethnography' paper presented at the Digital World Research Centre - The University of Surrey, Guildford, 19 November 2018. 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Other audiences
Results and Impact 'Journeys Between Print & Screen and Beyond: A Travel Media Ethnography' paper presented at the Digital World Research Centre - The University of Surrey, Guildford, 19 November 2018.
Year(s) Of Engagement Activity 2018
 
Description 'Touching upon Print & Screen: How media and technology reshape tourism' paper presented at The School of Hospitality and Tourism Management - The University of Surrey, Guildford, 4 December 2018. 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Other audiences
Results and Impact 'Touching upon Print & Screen: How media and technology reshape tourism' paper presented at The School of Hospitality and Tourism Management - The University of Surrey, Guildford, 4 December 2018.
Year(s) Of Engagement Activity 2018
 
Description Paper Magic Exhibition, COMM Museum, The Hague. 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact The project was invited to show its first 2G and 3G augmented paper travel guides from Chapter 2 of the 2nd Edition of the Bradt Travel Guide to Cornwall. The exhibition ran from July-December inclusive at the COMM Museum in the Hague which showcases developments in communications technology. As the books were not ready until the autumn of 2018 we supplied a video describing the project which was shown from July. David Frohlich and Radu Sporea later visited the museum on the 20th October to hand over the interactive books and give a scheduled demonstration to members of the public and invited media. This resulted in an article in the Dutch Press and request by the museum curator to use the technology in a 2019 exhibition of letters for which he wanted to make an augmented catalogue. We had a similar request to use the technology from the exhibition curator, and are working towards provision of a free cross-platform app to make available to these people and others who have requested the technology. The exhibition has now moved to the Public Library Amsterdam (OBA) from February to May 2019, and may go into the permanent collection at the COMM museum eventually as part of the history of communication.
Year(s) Of Engagement Activity 2018
URL https://www.surrey.ac.uk/news/showcasing-magic-paper
 
Description Presentation at 'Book in the Digital Age' Conference at Loughborough University, 24th October, 2018 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Other audiences
Results and Impact A 20-minute presentation and follow-up demonstration of the progress on new paper technologies from the Next Generation Paper Project was carried out to a mix of mostly academics from different fields and international universities. The presentation and demo sought to generate interest in the project and collect feedback from attendees, which it seemed to be successful in doing so. Many attendees asked detailed questions about the approach and had multiple recommendations for its utilisations in other contexts beyond Travel and Tourism-the sector that the project is currently focusing on.In fact, three publications were invited directly from this activity for the Journal of Electronic Publishing, Publishing History and The Writing Platform.
Year(s) Of Engagement Activity 2018
URL https://www.cphc.org.uk/events/2018/10/24/book-history-research-network-study-day-the-book-in-the-di...
 
Description Presentation at UGPN Annual Conference, University of Surrey 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact A 20-minute presentation and follow-up demonstration of the progress on new paper technologies from the Next Generation Paper Project was carried out to a mix of mostly academics from different fields and international universities. The presentation and demo sought to generate interest in the project and collect feedback from attendees, which it seemed to be successful in doing so. Many attendees asked detailed questions about the approach and had multiple recommendations for its utilisations in other contexts beyond Travel and Tourism-the sector that the project is currently focusing on.
Year(s) Of Engagement Activity 2018
URL http://www.ugpn.org/news/stories/172580_dates_announced_for_ugpn_annual_conference_2018.htm
 
Description Research Staff Celebration Week, Presentation, University of Surrey 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Other audiences
Results and Impact A presentation of the design challenges overcome in the Next Generation Paper project with a follow-up demonstration of the technology to audience members. Audience members included visitors from other universities. They gave value feedback on potential next stages for the project and the development of the technology.
Year(s) Of Engagement Activity 2018
URL https://www.surrey.ac.uk/events/20181119-research-staff-celebration-week-19-23-november-2018