Exaggeration, cohesion, and fragmentation in on-line forums

On-line forums can support the formation of social communities with shared interests and needs. They can also have a negative side if groups of users support each other in divisive attitudes or false beliefs. The social fragmentation resulting from these so-called echo-chamber effects has been identified as an engine behind the rise of violence and extremism, political gridlock, and decreases in social mobility. This project is motivated by the observation that echo-chamber effects involve a gradual shift from more moderate language to more extreme language. Further, damage repair is difficult when extreme social fragmentation has already occurred. The ability to use patterns in on-line language for early detection of on-line social fragmentation would thus be a major breakthrough in supporting earlier, and more effective, intervention against harmful trends in on-line forums.

We have identified two major challenges in creating this capability. First, current NLP methods are poor at understanding expressions whose meaning is a degree on a scale, such as a scale defined on the dimensions of cost, quality, honesty, or performance. For example, "rather racist", "really racist", and "incredibly racist" express different degrees of disapproval, but such differences are not adequately captured by current algorithms. This limitation is central to our problem, because echo-chamber effects often involve incremental exaggerations of factual claims, emotions, or attitudes. The second challenge results from the fact that methods for using linguistic content in the analysis of social behaviour are limited. While much research has uncovered systematic associations between word choices and social groups, very little has addressed relationships between linguistic inferences and social trends. However, tracking the gradual shifts towards semantic extremes in echo-chamber effects requires making certain linguistic inferences. This is because inferring which underlying dimension of meaning is relevant in any specific case critically depends on information about who is talking and what they are talking about. For example, "Liverpool is far better" might to relate a scale of cultural excellence in a discussion amongst music fans, but to a scale of costs amongst people who are discussing housing. A fundamental advance in the methodology for combining linguistic and social information is thus needed to characterise echo-chamber effects on-line and make predictions about risks of future fragmentation.

The project is a new collaboration between an experimental and computational linguist (the PI) and an expert in machine learning and social network analysis (the Co-I). Its components integrate the expertise of both collaborators. Advanced text-mining and data analytics will be used to generate the materials for a large-scale and experimentally normed data set of scalar expressions, using archives of the popular on-line forum Reddit. No normed data set of this type exists, and it will provide the training and test materials needed to develop and evaluate new algorithms. Using a modular work plan, the project team will first develop and validate separate algorithms to assess and predict the meanings of scalar expressions, and the level of fragmentation in the social network of Reddit users. These components will then be integrated using advanced graph-based machine learning methods. The primary outcome of the project will be a software package that will facilitate the work of on-line moderators by flagging subReddits or threads that display early stages of echo-chamber effects. The normed data set will also be extremely valuable for improving NLP applications that require nontrivial semantic inference, such as sentiment analysis, chatbots, and question-answering systems. More generally, the project is a demonstration project for advanced methodology in processing linguistic meaning in relation to social relationships and human behaviour.

The impact of the project will be realised in the following ways:

Academic impact:
If successful, the project will play a key role in the development of novel techniques for semantic inference, an important and open research area in statistical NLP where current methods are largely insufficient. The normed data set and software implementation developed in the project will serve as standard benchmark for further studies in NLP, and contribute to the promotion of reproducible research and open science. The project will further benefit research on text and document analysis, language modelling, and computational linguistics.

Beyond NLP and text-related analysis, the project contributes to a number of emerging research areas. One the one hand, the handling of heterogeneous and network-structured data from on-line discussion forums enables new areas of application for state-of-the-art data analysis methods, e.g., graph signal processing and geometric deep learning. On the other hand, the utilisation of linguistic content in the analysis of social behaviour is an excellent example of and opens new possibilities in the new fields of computational social science and social data science.

Industry impact:
Challenges in statistical NLP are pervasive in today's commercial systems. Reddit, Facebook, and Google all face similar challenges in monitoring and analysing on-line materials, and will benefit from the advanced capabilities in this domain that our project will provide. The research on semantic inference will benefit commercial dialogue systems and chatbots developed by corporations such as Apple and Amazon, while the consequent improvements in sentiment analysis of text may prove valuable for marketing or recommender systems developed at Amazon and Netflix.

Societal impact:
Segregation and inequality are widespread phenomena that challenges societies across the globe. The project would lead to better understanding of social fragmentation and the echo-chamber effects, in particular in the on-line setting in the digital age, which may help mitigate negative economic outcomes such as lower social mobility and higher inequality. Timely prediction of trends of social fragmentation also provides opportunities for early intervention to break potentially harmful echo chambers, and promote social cohesion both on-line and off-line.

Competitiveness of UK research:
The project focuses on the interplay between statistical NLP and machine learning/artificial intelligence, two priority areas recognised by the EPSRC and UKRI. Furthermore, the combination of ideas in NLP, machine learning, network science, and social sciences is perfectly aligned with the UK`s emphasis on cross-disciplinary research. The project further strengthens these areas for which the UK is already well-known internationally. Through the EPSRC CDTs and Oxford post-graduate programmes that the PI and Co-I are involved in, it will help to ensure that the UK continues playing a leading role in these key areas through the training of next-generation researchers and policy-makers.