Exaggeration, cohesion, and fragmentation in on-line forums

Lead Research Organisation: University of Oxford
Department Name: Engineering Science


On-line forums can support the formation of social communities with shared interests and needs. They can also have a negative side if groups of users support each other in divisive attitudes or false beliefs. The social fragmentation resulting from these so-called echo-chamber effects has been identified as an engine behind the rise of violence and extremism, political gridlock, and decreases in social mobility. This project is motivated by the observation that echo-chamber effects involve a gradual shift from more moderate language to more extreme language. Further, damage repair is difficult when extreme social fragmentation has already occurred. The ability to use patterns in on-line language for early detection of on-line social fragmentation would thus be a major breakthrough in supporting earlier, and more effective, intervention against harmful trends in on-line forums.

We have identified two major challenges in creating this capability. First, current NLP methods are poor at understanding expressions whose meaning is a degree on a scale, such as a scale defined on the dimensions of cost, quality, honesty, or performance. For example, "rather racist", "really racist", and "incredibly racist" express different degrees of disapproval, but such differences are not adequately captured by current algorithms. This limitation is central to our problem, because echo-chamber effects often involve incremental exaggerations of factual claims, emotions, or attitudes. The second challenge results from the fact that methods for using linguistic content in the analysis of social behaviour are limited. While much research has uncovered systematic associations between word choices and social groups, very little has addressed relationships between linguistic inferences and social trends. However, tracking the gradual shifts towards semantic extremes in echo-chamber effects requires making certain linguistic inferences. This is because inferring which underlying dimension of meaning is relevant in any specific case critically depends on information about who is talking and what they are talking about. For example, "Liverpool is far better" might to relate a scale of cultural excellence in a discussion amongst music fans, but to a scale of costs amongst people who are discussing housing. A fundamental advance in the methodology for combining linguistic and social information is thus needed to characterise echo-chamber effects on-line and make predictions about risks of future fragmentation.

The project is a new collaboration between an experimental and computational linguist (the PI) and an expert in machine learning and social network analysis (the Co-I). Its components integrate the expertise of both collaborators. Advanced text-mining and data analytics will be used to generate the materials for a large-scale and experimentally normed data set of scalar expressions, using archives of the popular on-line forum Reddit. No normed data set of this type exists, and it will provide the training and test materials needed to develop and evaluate new algorithms. Using a modular work plan, the project team will first develop and validate separate algorithms to assess and predict the meanings of scalar expressions, and the level of fragmentation in the social network of Reddit users. These components will then be integrated using advanced graph-based machine learning methods. The primary outcome of the project will be a software package that will facilitate the work of on-line moderators by flagging subReddits or threads that display early stages of echo-chamber effects. The normed data set will also be extremely valuable for improving NLP applications that require nontrivial semantic inference, such as sentiment analysis, chatbots, and question-answering systems. More generally, the project is a demonstration project for advanced methodology in processing linguistic meaning in relation to social relationships and human behaviour.

Planned Impact

The impact of the project will be realised in the following ways:

Academic impact:
If successful, the project will play a key role in the development of novel techniques for semantic inference, an important and open research area in statistical NLP where current methods are largely insufficient. The normed data set and software implementation developed in the project will serve as standard benchmark for further studies in NLP, and contribute to the promotion of reproducible research and open science. The project will further benefit research on text and document analysis, language modelling, and computational linguistics.

Beyond NLP and text-related analysis, the project contributes to a number of emerging research areas. One the one hand, the handling of heterogeneous and network-structured data from on-line discussion forums enables new areas of application for state-of-the-art data analysis methods, e.g., graph signal processing and geometric deep learning. On the other hand, the utilisation of linguistic content in the analysis of social behaviour is an excellent example of and opens new possibilities in the new fields of computational social science and social data science.

Industry impact:
Challenges in statistical NLP are pervasive in today's commercial systems. Reddit, Facebook, and Google all face similar challenges in monitoring and analysing on-line materials, and will benefit from the advanced capabilities in this domain that our project will provide. The research on semantic inference will benefit commercial dialogue systems and chatbots developed by corporations such as Apple and Amazon, while the consequent improvements in sentiment analysis of text may prove valuable for marketing or recommender systems developed at Amazon and Netflix.

Societal impact:
Segregation and inequality are widespread phenomena that challenges societies across the globe. The project would lead to better understanding of social fragmentation and the echo-chamber effects, in particular in the on-line setting in the digital age, which may help mitigate negative economic outcomes such as lower social mobility and higher inequality. Timely prediction of trends of social fragmentation also provides opportunities for early intervention to break potentially harmful echo chambers, and promote social cohesion both on-line and off-line.

Competitiveness of UK research:
The project focuses on the interplay between statistical NLP and machine learning/artificial intelligence, two priority areas recognised by the EPSRC and UKRI. Furthermore, the combination of ideas in NLP, machine learning, network science, and social sciences is perfectly aligned with the UK`s emphasis on cross-disciplinary research. The project further strengthens these areas for which the UK is already well-known internationally. Through the EPSRC CDTs and Oxford post-graduate programmes that the PI and Co-I are involved in, it will help to ensure that the UK continues playing a leading role in these key areas through the training of next-generation researchers and policy-makers.


10 25 50

publication icon
Drinkall F. (2022) Forecasting COVID-19 Caseloads Using Unsupervised Embedding Clusters of Social Media Posts in NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference

publication icon
Hofmann V. (2021) Dynamic contextualized word embeddings in ACL-IJCNLP 2021 - 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Proceedings of the Conference

publication icon
Hofmann V. (2021) Superbizarre is not superb: Derivational morphology improves BERT's interpretation of complex words in ACL-IJCNLP 2021 - 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Proceedings of the Conference

Description Large pretrained language models, such as ChatGPT, have impressive performance and are much in the news. One goal of the project is to assess the extent to which they capture aspects of meaning that are important for figuring out whether people are exagerating. By carefully probing the models, we have found that they understand this aspect of human language rather poorly. This is partly because they do not represent complex words accurately; for example, one of the most used models represents "superbizarre" as having a meaning close to "superb". Another reason is that they do not have good representations of adverbs such as "hardly", "quite", and "very", confusing all of these with "not".
Exploitation Route The outcomes so far could be put to use in developing better language models for answering questions, for helping moderators to identify toxic on-line posts, and other similar natural language processing tasks.
Sectors Digital/Communication/Information Technologies (including Software)

Description Röttger et al. (2021) identified very significant problems with a commercial hate speech moderation algorithm, the Two Hat SiftNinja algorithm. For a variety of functional types that were investigated, it had performance that was well below chance (both confusing hate speech with acceptable speech, and acceptable speech with hate speech). This resulted in Two Hat immediately taking its project off the market.
First Year Of Impact 2021
Sector Digital/Communication/Information Technologies (including Software)
Impact Types Societal,Policy & public services

Description Grand Union Doctoral Training Program.
Amount £110,420 (GBP)
Organisation Economic and Social Research Council 
Sector Public
Country United Kingdom
Start 09/2021 
End 09/2029
Title HateCheck: Functional Tests for Hate Speech Detection Models 
Description Detecting online hate is a difficult task that even state-of-the-art models struggle with. Typically, hate speech detection models are evaluated by measuring their performance on held-out test data using metrics such as accuracy and F1 score. However, this approach makes it difficult to identify specific model weak points. It also risks overestimating generalisable model performance due to increasingly well-evidenced systematic gaps and biases in hate speech datasets. To enable more targeted diagnostic insights, we introduce HateCheck, a suite of functional tests for hate speech detection models. We specify 29 model functionalities motivated by a review of previous research and a series of interviews with civil society stakeholders. We craft test cases for each functionality and validate their quality through a structured annotation process. To illustrate HateCheck's utility, we test near-state-of-the-art transformer models as well as two popular commercial models, revealing critical model weaknesses. 
Type Of Material Database/Collection of data 
Year Produced 2020 
Provided To Others? Yes  
Impact This normed dataset for diagnosing problems with English hate speech detection algorithms revealed such severe problems with the TwoHat algorithm that is was taken off the market the next day after the paper appeared. It also resulted in a contract from Google for a multi-lingual counterpart to the English version. Please note that support from various other sources (including the Turing Institute and the Oxford Internet Institute) was important to the success of this multi-disciplinary project, and so the credit should be shared. 
URL https://arxiv.org/abs/2012.15606
Title The Reddit Politosphere 
Description This is the largest scale database of on-line political discussion now available. Unlike other databases relating to political discourse on Reddit, it was not created by selecting particular subreddits based on the subreddit name or the political orientation. Instead, all 600+ subreddits that are dominated by political discussion, as determined by objective natural language processing methods, are included. In addition, the database includes a network of relations amongst the subreddits. It is based on the overlap in contributors to the subreddits, pruned to the most significant connections using an up-to-date network sparsification method. The dataset is also enriched with various types of useful metadata. 
Type Of Material Database/Collection of data 
Year Produced 2022 
Provided To Others? Yes  
Impact None to date -- has only been available for 2 months. 
URL https://zenodo.org/record/5851729