World Futures: Multimodal Viewpoint Construction by Russian International Media

Lead Research Organisation: University of Oxford
Department Name: Area Studies


Discussions of futures have never been more important and challenging than now when the world finds itself fighting COVID-19. People engage in conversations about what our lives will look like after the pandemic, and so do the media.

The media often talk about futures to frame the way we think, and the media funded by the Russian State use viewpoint construction rooted in the depiction of futures as a subtle but powerful approach to manipulate and influence public opinion.

Viewpoint construction techniques are not just verbal but multimodal: they combine words, prosodic cues such as intonation and timing, gesture, and movement, and other non-verbal elements. They rely heavily on culture, history, and other contextual knowledge, and if used for manipulative purposes, lead the viewer to draw a false conclusion, make an assessment, or come to an opinion that is beneficial to the hostile actor, without being aware that this is happening.

In the true spirit of the digital humanities, our 2-year project will fuse cognitive and corpus-driven/based analyses of language, prosody, and gesture with area studies (regional knowledge: culture, literature, history, and society), while leveraging latest developments in machine learning, natural language processing (NLP) and computer vision (CV) to tackle the problem at scale.

We will combine our multimodal analysis of Russian international media broadcasts (English and Russian) with the analysis of audiences' comments on social media to open a window onto viewpoint construction in the audiences' minds and thus provide additional validity to our linguistic analysis. Our project will break new ground in cognitive and corpus linguistics to significantly increase the overall reliability of large media data analysis currently used in linguistics as well as other humanities and the social sciences.

Our approach will help answer questions posed by academic researchers and Western policymakers in relation to future information threats and mitigations, as applied to Russia as a region of strategic importance. Ultimately, the project will go beyond the applied linguistics approach to disinformation analysis. It will enable researchers to test multimodal patterns found in a systematic way, thereby addressing one of the biggest challenges linguists face.

Furthermore, the project will contribute to answering such big questions in multimodal research in theoretical linguistics (semantics, syntax, pragmatics, comparative) as:

1) How are meanings constructed multimodally and how are multimodal meanings perceived?

2) How is the construction of meanings distributed across gestural, prosodic, and verbal modes? Are the "redundant" modes of gesture and prosody really redundant?

3) How do languages with quite distinct grammatical and phonetic properties, rooted in quite diverse historical and cultural backgrounds, realise the ideas of time and space multimodally while pursuing the same overarching communicative and pragmatic goals?


10 25 50
Description At this stage in our project we already have some preliminary findings and results stemming both from our manual annotation of data and subsequent manual analysis. Our manual analysis has been informed by our progress on the automation of analysis of speech and gesture. And our work on automation of the said analysis has been in turn informed by the insights gained through our manual analysis. We have presented some of our findings at the conference of the international society of gesture studies in July 2022, and received valuable feedback. Furthermore we have submitted our findings to date for presentation at three major international conferences in the fields of cognitive linguistics, contrastive linguistics, and gesture studies in July-September 2023.
Exploitation Route Upon the completion of our project in 2024 our published findings, our datasets, and the software we develop or customise will allow researchers worldwide to conduct analysis of multimodal big data faster and more efficiently. They will also allow our non-academic partner to conduct analysis of disinformation specifically relying on the depictions of futures at scale and speed.
Sectors Aerospace, Defence and Marine

Title Multimodal Corpus Pipeline 
Description We developed a pipeline specifically for the type of media data collected within our project, including NLP, gesture detection and biometric clustering. 
Type Of Material Improvements to research infrastructure 
Year Produced 2023 
Provided To Others? No  
Impact The pipeline significantly reduces time needed for manual annotation hence reducing the resources (labour and money) needed for analysis. 
Description Collaboration with Dr Siddharth, University of Edinburgh 
Organisation University of Edinburgh
Department School of Informatics Edinburgh
Country United Kingdom 
Sector Academic/University 
PI Contribution Our team provides expertise on qualitative analysis of gesture in communication as well as insights and annotated data resulted from our manual analysis.
Collaborator Contribution Dr Siddharth provides expertise on computer vision and machine learning guiding the machine learning engineer hired by our research team on the German side.
Impact Outcomes: various models for detection of gesture are being developed and evaluated.
Start Year 2022
Description Collaboration with Prof. Kita, University of Warwick 
Organisation University of Warwick
Department Department of Psychology
Country United Kingdom 
Sector Academic/University 
PI Contribution Our team's findings have contributed to the development of theory and methods in gesture studies (speech-gesture interaction).
Collaborator Contribution Professor Sotaro Kita has contributed to discussions on theory and methods developing in gesture studies, including insights stemming from the field of psychology.
Impact Our discussions have contributed to reformulating an aspect of our research design in line with psycholinguistics theory and methods.
Start Year 2022
Description Policy Engagement 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Policymakers/politicians
Results and Impact We have been engaging in a constant dialogue with our UK non-academic partner as indicated in our grant application via email, phone conversations and participation in the project event in June 2022.
Year(s) Of Engagement Activity 2022,2023