Reel Lives: personal documentaries constructed from digital data.

Lead Research Organisation: Northumbria University
Department Name: Fac of Health and Life Sciences

Abstract

New social media has led to an explosion in personal digital data that encompasses both those expressions of self chosen by the individual as well as reflections of self provided by other, third parties. The resulting Digital Personhood (DP) data is complex and for many users it is too easy to become lost in the mire of digital data. This is unfortunate as these digital representations of self are not only valuable to the individual as a means of self-reflection, but also have great information value as a community, business and social policy resource. In this proposal, we argue that the ability to represent and edit personal digital data in a meaningful form is crucial to the empowerment of individuals and communities and that it could also generate significant benefit to government and business. We would also argue that, this 'meaningful form' should include a filmic, narrative structure that is easy to comprehend and edit and that allows for a simply comparison of one digital life with another. In short, we propose building searchable, personal film-like documentaries (that we would term 'Reels') from existing, digital data.

To build the Reels, we must first develop the means of capturing online life logs, events and associated data. A variety of means already exist to do this, but we must ensure that we are able to take data that is distributed over multiple social networks and integrate it into a single profile that captures a users interests and experiences, social networks, behaviour, personality and mood states at various points in their own personal timeline. These profiles can then provide us with rich representations and understanding of users' life events and contexts at different points in time, which we can process further to generate video storylines and narrative. A simple time-line structure is insufficient in and of itself to create a meaningful narrative, and a crucial technical issue here regards the extent to which existing digital material can be matched to existing, known narrative structures in order to make sense of a diffuse landscape of events and images. With the right narrative structures in place, the objects, images and events can then be brought to life.

Our aim is to develop systems that can convert the available profile data into meaningful personal documentaries (Reels) that individuals can either edit themselves or that can be turned over to film and media professionals to create Reels of intrinsic artistic value. Crucial to this project is the fact that each Reel is completely searchable, because it is generated from underlying semantic data. This allows for the direct comparison of one individual digital life with another or facilitates the collection and comparison of digitally similar Reels that allow enhanced understanding of communities and the citizens within them. We propose a significant period of user and stakeholder engagement with business and community leaders to understand the value of such comparisons.

Planned Impact

We anticipate impact on the citizen, communities, business and government in terms of new insights for the development of social policy.
At the citizen level, the Reels would constitute powerful tools for personal reflection and understanding of our digital personas - citizens will be better able to see their Digital Personhood (DP) as a coherent narrative - allowing them to see themselves as others may see them. This new insight is coupled with opportunities to directly edit this data. Editing of DP data is extremely difficult at present - due to the fragmentation of this digital data and its distribution across several systems, accounts, and time windows. However editing a reel of film is a conceptually simple process. Our Reels allow this film editing as a tool to empower users to take control of their digital lives, for example, setting disclosure preferences in a seamless form.

This editing process also offers a new business model for the creative industries, whereby individuals who lack the time or the interest in developing and editing their own Reels could turn to creative media companies who might offer this service.

Beyond the citizen and the business of individual Reel creation, the power of the Reels lie in the capacity to establish collections of Reels that permit comparison of one individual with another, or one community with another. This is possible because each Reel is directly generated by underlying semantic data which means that every Reel is completely searchable. It becomes easy, then, to develop a searched collection that displays, say, the lives of young offenders and compares and contrasts them with the lives of others from similar backgrounds and initially similar interests - capturing the points at which their paths diverged. The Reel Collections could therefore be used by community leaders to gain greater insight into the individual factors that underpin community change or tension and there could be an important role for Reels in the development of social policy. We will work with communities to understand this opportunity, but also propose a collaboration with those working in peace and reconciliation in order to develop the use of the Reels in that specific context.

Engagement and impact will be managed via an advisory group and will be undertaken in terms of (i) stakeholder engagement with the creative industries, (ii) stakeholder engagement with local government, community groups and the Home Office and (iii) citizen engagement through participatory design and also more widely through large-scale public understanding and engagement activities. These plans are described in more detail in the pathways to impact document.

Our impact plans are based on the existing expertise each applicant brings to ensuring the production of outputs relevant to varied users outside academia. This expertise is documented in the case for support and attached CVs, but covers film-making and marketing (Kerrigan), participatory design and stakeholder engagement around identity management (Briggs), technological developments and business opportunity awareness underpinning the development of narrative (Aylett) and the commercial and social value of semantic web data and search algorithms in social media (Alani).

Publications

10 25 50
 
Title Six short films constructed entirely from existing social media data 
Description We have a series of six short films constructed entirely from social media data. The six films were showcased at the National Media Museum on 21st October, 2014. The films are described here: The Public Life of Manick G Directed by Meghan Horvath If social media made a noise, what would it sound like? Meghan's instinct when creating The Public Life of Manick G. was to prioritise sound. By bringing to life the conversations and debate initiated by the film's main character Manick, the aim was to explore the contrast between noise and meaning as well as the boundary between our personal and public personas in the online sphere. As a filmmaker, Meghan directs both fiction and documentaries. She is inspired by stories that challenge her to uncover surprising angles within familiar issues, especially when they provoke questions about identity. CUMULUS Directed by Christopher Ian Smith CUMULUS is an experimental documentary project focused on the musician Imogen Heap. Using only data and content on social media created by Imogen and her fans, CUMULUS crafts a narrative that explores Imogen's background and personality, the unique relationship she has with her fans, and her innovative musical practice. This includes her recent development of the MI.MU musical gloves that - she sees as an extension of herself into the digital space though wearable technology. As an early adopter of new and emerging communication channels, Imogen's digital footprint is varied and far-reaching as her digital personality grows through the interaction of her fans. Her work and influence also continues to evolve as simulacrum in the tributes created by her fans, generating multiple versions of Imogen and her work in the digital space. Everyone is Waiting for Something to Happen Directed by Emma Calder Prior to being diagnosed with bowel cancer Richard Wright, an artist/animator, had a social media persona that was characterised by annoying and anarchic humour. After initially feeling unable to communicate at all, his general healing process became entwined with the resurrection of his social media presence, helped by an obsessive regime of cooking and baking. The film uses character and abstract animation, created from collages of Richards photos and quotes posted on social media including a fifty two page blog which he wrote for a bowel cancer forum and interviews with Richard talking about his posts. The result is a humorous and visually stimulating film that explores one creative individuals strategy for coping with a life threatening illness and how he was able to share this with his social media community. Emma has worked as an Animation Director, Artist, Graphic Designer and Lecturer. She has written, designed and illustrated books, for both adults and children. This is her second documentary film. Peter Tatchell? Directed by Peter Fraser This film explores the public life and personal motivations of human rights activist Peter Tatchell exclusively through material gathered from his social media profiles. Peter is best known for his highly influential work with LGBT movements but has campaigned on a range of human rights and political issues since 1967 throughout a high profile and controversial career. Peter lists his key political inspirations as 'Mahatma Gandhi, Sylvia Pankurst, Martin Luther King and, to some extent, Malcolm X and Rosa Luxembourg.' As an outspoken and unstinting political activist Peter employs social media extensively as a platform for his views and campaigns. Peter is a documentary filmmaker who has worked on a wide range of subjects internationally. He is particularly interested in how social media mediates such a politicised personality. The Baristacrat Directed by Gus Melton Representation through social media has always fascinated Gus. The ever-evolving landscape of the Internet provides many tools to express ourselves. From the candid to the enigmatic, these tools can be manipulated in order to control an image. Gus wanted to explore what was possible in taking a subject that had minimal, yet ever increasing, online content and flesh out the alter ego being presented through the different modes being used. Whether it be a pitch or a showreel, Gus is frequently tasked with carving a narrative out of very little content. Despite this, threading the minimalist elements of the blog together still came as a challenge. Keeping the pace punchy and the tone light was crucial to ensure Gus made the piece as exciting as possible whilst staying true to the aesthetics of the original blog. By mirroring the quirks of the writing with a well-weighted score, Gus tried to display the alter ego's full character in order to round off a complete package of consolidated social media streams. "Alice" - A Social Media Journey Directed by Louise Cooke Is there more to a person than collected words and images? A very sociable person, Alice has used her many social media accounts as a visual scrapbook of her life - the highs and lows, the happiness and heartache. In her own words she 'uses Facebook like a diary'. As a filmmaker, Louise is preoccupied with portraying women in a different, more real way, to how they're normally portrayed in the media. Making this film has allowed Louise to explore themes of interest in a new format. Alice goes on a journey of change but what role did social media play in helping Alice enable those changes? Are we how we say we are or do we say what we want to be? 
Type Of Art Artwork 
Year Produced 2014 
Impact We are intending to support the entry of these films into competition and promote discussion of the techniques underpinning them. 
URL http://reellives.net/competition/
 
Description During the first year we have assessed the relevance of different social media outlets for the REEL Lives project, captured user requirements and opportunities for personal biographies, commissioned six high-quality filmed biographies constructed from existing social media data, with accompanying interviews of film-makers and subjects. We have developed an initial prototype for the narrative generation module and have started to produce narratives based on triptychs. Finally we developed a web interface that connects to certain social media outlets and that facilitates the process of searching for relevant social media content.
During the later stages of the project we have developed a range of new techniques for the automated analysis of social media data involving new ways of classifying social media posts to build up new techniques for event recognition.
Our recent focus for this period has been looking at creating social media post classifiers, with the goal of identifying posts that are about events. Several classifiers were trained on Twitter data, focusing on five life events identified in psychological literature: Getting Married, Having Children, Starting School, Death of a Parent, and First Love. We also considered a binary classifier: about an event or not. Our results showed good performance when taking into context the content of the post (i.e. the words). During our study, we noticed the importance in differentiating between posts about the theme of an event, and posts that express an event taking place. To this end, we extended our classifiers with event-patterns, extracted from syntactic-semantic graphs that we generate for each post. This significantly increases our ability to correctly identify the occurrence in events in post streams.

Associated with this work, we have been investigating the reminiscence bump as a content retrieval/recommendation problem based on the digital archives of personal lives, created by users in social platforms. To better understand which memories or events users will want to recollect from their social profiles, we are following a semantic approach based on the following steps:
Specifically, we aim to investigate which type of content people share in their social profiles (hobbies, family pictures, work-related information, etc.) and weather we can identify patterns among the content shared and the content sharing practices. To investigate this issue we propose to generate a semantic layer linking users with their produced content items (posts). Content items (posts) are associated to the semantic layer (entities and relations) via weighted semantic annotations. To extract these annotations we will make use of existing Information Extraction technologies (such as DBPediaSpotlight or AlchemyAPI). On the other hand, users are linked to the semantic layer via weighted semantic preferences, where user profiles are created based on the entities and relations that users mention within their posts. The creation of this semantic layer will help us to investigate which concepts are recurrent within the user profiles over time as well as which concepts are more unusual. This semantic layer will be complemented by additional features that can better help us to interpret the relevance of memories for each particular user. These features include the sentiment and emotions expressed within the posts, the language usage/directives used by the user (for example, if the post refer to the individual (I/me) or to a group (we/us)), or the popularity generated by the post (i.e., the attention generated by the post in other users and expressed via likes, shares, comments, retweets, etc.).

In addition, we have developed software for the automatic extraction of triptychs from social media data and have conducted an evaluation of those automatically generated triptychs by the end user. This work has been presented at 2015 and 2016 conferences on human-computer interaction and has been submitted to the 2017 Design of Information Systems Conference and to the Journal of Human Computer Studies. A system for triptych generation has been developed and evaluated.

Finally, following our film work, we have moved to explore the ways in which social media can represent communities - extracting social media profiles from people who are engaged in the process of joining a new community and determining both the meaning of social media for the individual, but also the key semantic terms that will underpin community based activity. To this end we have developed a novel method of using pinterest to 'pin' social media and then use these pinboards as probes to interview people around their own perceptions of their social media use. This novel technique has been used to explore identity transitions in young people moving to university and a paper has been accepted in the journal Computers in Human Behaviour.
Exploitation Route The creative industries will be interested in the toolset that we are developing, but can also use our films as inspiration for the different ways in which social media data can be repurposed. Our user studies are generating new insights into digital personhood, particularly around privacy that could be used to inform citizen rights groups.

Recently, we were approached by the IBM Watson team, who expressed their interest in our personal-event extraction work. We held one teleconference with the IBM team so far, and provided them with training data and papers to examine before starting a formal collaboration process.

Other EU projects have been developed on the back of findings from this award. Specifically, the event-extraction will also be taken forward as part of an H2020 EU project that started in January 2016, called COMRADES. A core component of the new project is the extraction of personal disaster-related event during crises.

There has been interest in our triptych work following presentations at the Alan Turing Institute and at IFIP Interact 15 and this is likely to be the subject of a further funding bid in association with Bloomberg, New York . We have since publiched a paper on triptychs.

We held an international workshop (May 2016) entitled 'my life on film' co-organised with colleagues at Yahoo to explore the use of social media data in personal biographies. We have interest from social media professionals and are following up a contact from YoTi - around a digital identity app.

We have been invited to workshops led by the Home Office and the EU High Level Scientific Advisors to discuss issues around managing digital identity and consider our findings in the context of national and international policy development.
Sectors Creative Economy,Digital/Communication/Information Technologies (including Software),Leisure Activities, including Sports, Recreation and Tourism,Culture, Heritage, Museums and Collections

URL http://reellives.net/
 
Description At the end of our first year, the major impact has been on the arts and creative industries, with the launch of our Film Competition and the premiere, in October 2014, of six short films constructed entirely from digital data. During 2015 There were two public screenings of the six films; National Media Museum in October 2014, and an ESRC Festival of Social Science funded screening at the Midlands Arts Centre in November 2015. In addition, individual films were screened at (i) Wimbledon International Short Film Festival; (ii) Sheffield DocFest; (iii) The SoCal Creartive & Innovative Film Festival; (iv) Edinburgh International Film Festival and (v) London International Animation Festival and the Direct Short and Documentary Film Festival, where 'Alice' was a winner (Best Director). School-based workshops have drawn on these films in order to engage young women around the concept of 'digital safeguarding'. The aim is to produce a drama package of materials to be rolled out in schools, in association with third sector organisation 'The Forge' and to engage educators more widely in the issue of digital personhood. In 2017 we conducted a number of workshops with university students on the themes of social media in support of 'transitioning to university' and made our findings known to university administrators charged with induction. In 2017 we made a presentation of some of the Reel Lives findings to a Joint meeting organised by the Royal Society and the Home Office (CAST). This meeting on 'Assured Identities' was attended by various branches of UK government and industry connected to identity. A follow up is planned with possible joint work with one of the industry partners. Two new EU research projects have been influenced by this work: COMRADES http://comrades-project.eu/ €2 million, Jan 2016-Dec 2018 Partners: OU (coordinator), Uni Agder (Norway), IHub (Kenya), Uni Sheffield, Government 2 You (Belgium) -- COMRADES reuses the work of ReelLives on events detection, and extends it by applying it to the domain of disaster management. TRIVALENT (no website yet) May 2017 - April 2020 €2.7 million -- applies ReelLives events tracking to the field of online radicalisation analysis. Partners: 21 EU partners, including several law enforcing authorities across several EU countries (Portugal, Latvia, Belgium, Spain, Italy) Briggs was invited to join the All-Party Parliamentary Group (APPG) on digital identity - seeking to influence government policy on digital identity.
First Year Of Impact 2014
Sector Creative Economy,Digital/Communication/Information Technologies (including Software),Education,Security and Diplomacy
Impact Types Cultural,Societal

 
Description Input to EU Scientific Advice Mechanism (SAM) - Workshop organised by the High Level's Group of Scientific Advisors to the European Commission
Geographic Reach Europe 
Policy Influence Type Gave evidence to a government review
 
Description Public Engagement with School Children with The Forge 
Organisation The Forge Ltd
Country United Kingdom 
Sector Private 
PI Contribution Working with school children to explore their perceptions of disclosure on social media and their social media profile - we provided research context
Collaborator Contribution The partners set up workshop with local school children
Impact Public engagement event, multi-disciplinary
Start Year 2015
 
Description Conference on Human-Computer Interaction (CHI) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Industry/Business
Results and Impact In 2015 we presented the triptych work to a mixed international audience
Year(s) Of Engagement Activity 2015,2016
 
Description Contribution to All Party Parliamentary Group on Digital Identity 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Policymakers/politicians
Results and Impact Discussion in APPG on government priorities for digital identity management
Year(s) Of Engagement Activity 2017,2018
 
Description Presentation as part of the Turing Institute's Data Science summits 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Industry/Business
Results and Impact This was a 'Data Science for Media Summit' organised by the Alan Turing Institute (the national institute for data science, with headquarters at the British Library, although this summit was in Edinburgh).
Year(s) Of Engagement Activity 2015
 
Description Screening at Edinburgh International Film Festival 
Form Of Engagement Activity A broadcast e.g. TV/radio/film/podcast (other than news/press)
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Screening of 'Everyone is waiting for something to happen' Film produced as part of the REELLIVES project
Year(s) Of Engagement Activity 2015
 
Description Screening at Sheffield Docfest 
Form Of Engagement Activity A broadcast e.g. TV/radio/film/podcast (other than news/press)
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Screening of 'The Public Life of Manick G' at Sheffield Docfest -
Year(s) Of Engagement Activity 2015
URL http://www.bfi.org.uk/news-opinion/news-bfi/features/preview-sheffield-docfest-2015
 
Description Screening at the London International Animation Festival 
Form Of Engagement Activity A broadcast e.g. TV/radio/film/podcast (other than news/press)
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Screening of 'Everyone is waiting for something to happen' at the London International Animation Festival
Year(s) Of Engagement Activity 2015
URL http://www.liaf.org.uk/2015/10/liaf-2015-animated-documentaries-15/
 
Description Theatre in Education event in Schools 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Schools
Results and Impact Theatre in Education events on digital safeguarding run in association with third sector organisation 'The Forge' and using film and materials from the Reel Lives project to produce student-led drama experiences
Year(s) Of Engagement Activity 2015,2016
 
Description Workshop with undergraduate students re: transition to university 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Undergraduate students
Results and Impact We designed technological probes which were informed by ReelLives work- the idea of 'YourStory'- a social media based archive of time at university. The workshop utilised this probe among others to understand how students manage the transition to university, dealing with moving away from home, making friends, and documenting their experiences online.
Year(s) Of Engagement Activity 2017
 
Description presentation to the Cyber Aware Industry Forum (27th February, 2018) 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Industry/Business
Results and Impact At the invitation-only event, we bring together organisations with a shared interest in encouraging consumers and small business owners to be more cyber secure, and discuss how best all sectors can work together to respond to the cyber threat.

A government led, industry forum with a focus on the theme of protecting our online identity. Cyber Aware presents the risks cyber crime poses to customer and employee online identities, insights into public perceptions in this area, and the steps being taken to tackle this threat. I gave an invited presentation, followed by a panel discussion chaired by Britain Thinks that explored how our online identity is evolving and the opportunities this presents to cyber criminals, and how we can educate the public on the importance of protecting their personal information online.
Year(s) Of Engagement Activity 2018