World Futures: Multimodal Viewpoint Construction by Russian International Media

Discussions of futures have never been more important and challenging than now when the world finds itself fighting COVID-19. People engage in conversations about what our lives will look like after the pandemic, and so do the media.

The media often talk about futures to frame the way we think, and the media funded by the Russian State use viewpoint construction rooted in the depiction of futures as a subtle but powerful approach to manipulate and influence public opinion.

Viewpoint construction techniques are not just verbal but multimodal: they combine words, prosodic cues such as intonation and timing, gesture, and movement, and other non-verbal elements. They rely heavily on culture, history, and other contextual knowledge, and if used for manipulative purposes, lead the viewer to draw a false conclusion, make an assessment, or come to an opinion that is beneficial to the hostile actor, without being aware that this is happening.

In the true spirit of the digital humanities, our 2-year project will fuse cognitive and corpus-driven/based analyses of language, prosody, and gesture with area studies (regional knowledge: culture, literature, history, and society), while leveraging latest developments in machine learning, natural language processing (NLP) and computer vision (CV) to tackle the problem at scale.

We will combine our multimodal analysis of Russian international media broadcasts (English and Russian) with the analysis of audiences' comments on social media to open a window onto viewpoint construction in the audiences' minds and thus provide additional validity to our linguistic analysis. Our project will break new ground in cognitive and corpus linguistics to significantly increase the overall reliability of large media data analysis currently used in linguistics as well as other humanities and the social sciences.

Our approach will help answer questions posed by academic researchers and Western policymakers in relation to future information threats and mitigations, as applied to Russia as a region of strategic importance. Ultimately, the project will go beyond the applied linguistics approach to disinformation analysis. It will enable researchers to test multimodal patterns found in a systematic way, thereby addressing one of the biggest challenges linguists face.

Furthermore, the project will contribute to answering such big questions in multimodal research in theoretical linguistics (semantics, syntax, pragmatics, comparative) as:

1) How are meanings constructed multimodally and how are multimodal meanings perceived?

2) How is the construction of meanings distributed across gestural, prosodic, and verbal modes? Are the "redundant" modes of gesture and prosody really redundant?

3) How do languages with quite distinct grammatical and phonetic properties, rooted in quite diverse historical and cultural backgrounds, realise the ideas of time and space multimodally while pursuing the same overarching communicative and pragmatic goals?