Exploiting narrative structure in the generation of audio description of video

We are interested in the task of semi-automatically generating audio
descriptions for video. Audio description is "an additional audio
commentary developed primarily to enable people who are blind or have
sight loss to access audiovisual content" (Ofcom UK Accessibility
Guidelines, 2024). Major UK broadcasters are legally required to audio
describe 10% of their programmes, and in line with policies to make more
digital content accessible, this is expected to expand in the near
future. A sizeable creative industry already exists to produce audio
description. A single show can take multiple days for a team to describe
- it is a skilled task that goes beyond identifying actions in the
current scene, as it draws not only the video, but also knowledge of the
script, characterisation and the overall narrative.

Audio description has received computational treatment from the computer
vision community. There exist systems that take short clips as input and
generate verbal descriptions. The state-of-the-art approach involves
encoding the visual frames with one neural network (the visual encoder)
and learning to decode into the verbal domain (with a large language
model). Such systems have been augmented with surrounding
dialogue/narration, other audio and external knowledge sources (e.g.,
knowledge of casting and images of the actors). These systems are a
major milestone for the task. But the resulting (stiched together) audio
description is not engaging. There is no sense of narrative encoded, a
central component to any story. This project aims to tackle this
problem. We ask: what data structures can narratives take such that they
are (a) learnable by automatic methods and (b) useful to the task of
generating audio description?

In this project, the novel engineering will be to develop
self-supervised methods that model video-form narrative. Possible
directions include operationalizing theoretical approaches to narrative
structure or modelling the causal relationships that build up a
narrative. This will serve as an efficient approach to encode narrative
for the task of generating audio description. They may also serve to
validate particular theories of narrative. Our ultimate goal is to
semi-automatically generate compelling, sensitive and perhaps even
personalized audio description.