Visual Narrative Grammar: Multimodal Storytelling

Lead Research Organisation: University of Edinburgh
Department Name: Sch of Informatics


Visual storytelling comes in diverse forms from a personal album of a wedding or holiday, to public forms such as political cartoons, comics, movies, and fine art. There has been a lot of recent related work on multimodality: For example labeling or describing photos, creating representations that combine language with visual representations, question answering, learning event sequences, summarising movies, or generating the explanations for classifications. Storytelling, though, provides a different challenge in terms of tracking the development of plot and character to create interest that goes beyond classification and description. This makes it both an interesting technical challenge and is relevant for the better understanding human storytelling.

There is a wide a variety of recent techniques and methods such as RNN, CNN, and Memory based Neural Networks, as well as more traditional script and story generation techniques, that may be usefully employed at a later stage. The initial work, however, will focus on a pilot project based on a visual grammar for Comic Books created by Neil Cohn. The visual grammar is a structure for understanding the plot development in Comics. The pilot will evaluate this model for understanding comics, and for performing computational tasks such as sequencing or predicting cloze endings. Later work will extend this to larger datasets, alternative visual mediums such as movies, and more sophisticated models.

Timeline: Pilot of Visual Grammar on (3 months), (3 - 12 months) extension of dataset should pilot be successful to further similar comic and graphic novel content which may involve further data collection, (Year 2) comparison and evaluation more sophisticated models for understanding and generation of visual stories, (Year 3) extensions to possible other media such as movies or personal archives. (Year 3 - 3.5) write up.


10 25 50

Studentship Projects

Project Reference Relationship Related To Start End Student Name
EP/N509644/1 01/10/2016 30/09/2021
1985013 Studentship EP/N509644/1 01/01/2018 30/06/2021 David Rae Wilmot
Description Suspense is a crucial ingredient of narrative fiction, making stories compelling and engaging readers. While there is a vast theoretical literature on suspense, it is computationally not well understood. We compare two ways for modelling suspense: Surprise, a backwards-looking measure of how unexpected the current state is given the story so far; and uncertainty reduction, a forward-looking and measure of how unexpected the continuation of the story is. Both can be computed either directly over story representations or over their probability distributions. We propose a hierarchical language model that encodes stories and computes surprise and uncertainty reduction. Evaluating it against short stories annotated with human suspense judgements, we find that uncertainty reduction over representations is the best predictor, resulting in near-human accuracy. We also show that uncertainty reduction can be used to predict suspenseful events in movie synopses.
Exploitation Route The above is from a paper abstract that has to receive final acceptance and will be added when it is. The main practical outcome is a novel approach to understanding concepts of suspense and surprise in narrative text: This has a number of practical continuations including in improving neural text generation, use in a writing assistant, or as a textual analysis tool.
Sectors Creative Economy,Digital/Communication/Information Technologies (including Software),Culture, Heritage, Museums and Collections

Description We Need To Talk About AI Panel Discussion 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Other audiences
Results and Impact Primary organiser of this ongoing public engagement series: We Need To Talk About AI

We need to talk about AI. In fact, we are already talking about it: artificial intelligence and technology are a massive part of our daily lives. But the public debate is narrow and one-sided. On one hand, we have tech companies and investors overhyping technological advances. On the other - apocalyptic predictions: social media are feared for their interference with elections and shaping the public opinion to an extent never seen before and automation is rumoured to be on the way to replace all if not most of the jobs or even the humanity altogether.

However, this is all speculation and we need to talk about what the real issues surrounding AI are as well as what possibilities for the future it offers.

EdIntelligence are hosting a series of public discussions inviting insightful speakers from across disciplines including informatics, humanities, and social sciences. We aim to have an informed debate with a variety of perspectives on the important topics concerning advances in AI and modern technology.
Year(s) Of Engagement Activity 2019,2020