Visual Narrative Grammar: Multimodal Storytelling

Visual storytelling comes in diverse forms from a personal album of a wedding or holiday, to public forms such as political cartoons, comics, movies, and fine art. There has been a lot of recent related work on multimodality: For example labeling or describing photos, creating representations that combine language with visual representations, question answering, learning event sequences, summarising movies, or generating the explanations for classifications. Storytelling, though, provides a different challenge in terms of tracking the development of plot and character to create interest that goes beyond classification and description. This makes it both an interesting technical challenge and is relevant for the better understanding human storytelling.

There is a wide a variety of recent techniques and methods such as RNN, CNN, and Memory based Neural Networks, as well as more traditional script and story generation techniques, that may be usefully employed at a later stage. The initial work, however, will focus on a pilot project based on a visual grammar for Comic Books created by Neil Cohn. The visual grammar is a structure for understanding the plot development in Comics. The pilot will evaluate this model for understanding comics, and for performing computational tasks such as sequencing or predicting cloze endings. Later work will extend this to larger datasets, alternative visual mediums such as movies, and more sophisticated models.

Timeline: Pilot of Visual Grammar on (3 months), (3 - 12 months) extension of dataset should pilot be successful to further similar comic and graphic novel content which may involve further data collection, (Year 2) comparison and evaluation more sophisticated models for understanding and generation of visual stories, (Year 3) extensions to possible other media such as movies or personal archives. (Year 3 - 3.5) write up.