Equally Safe Online
Lead Research Organisation:
Heriot-Watt University
Department Name: School of Mathematical and Computer Sciences
Abstract
We address the timely topic of online gender-based violence (GBV): Almost 1 in every 2 women and non-binary people (46%) reported experiencing online abuse since the beginning of COVID-19 (Glitch report, 2020). Our aim is to create 'equally safe' online spaces by prevention, intervention and support for online GBV through the development of advanced Machine Learning algorithms.
In contrast to previous research in this area, our team will include experts on GBV and online harassment to ensure that we frame the problem in a way which is most helpful to victims/survivors of GBV. In other words, we not only focus on *how* to automatically detect online abuse, but also re-think *what* it is we need to detect, how we can *support* the victims and how to *prevent* online GBV through promoting digital citizenship (i.e. prevention and intervention aimed at perpetrators and bystanders).
Our methodology is based on the Scottish Government's Equally Safe strategy and implements a Responsible Innovation Approach in a close collaboration with 3rd sector charities with a long-standing track record in this domain. Our proposal aims to create the following impacts:
1. Create longterm technical solutions to support safer online spaces, including advanced abuse detection tools, tools to automatically generate counter narratives aimed at perpetrators and bystanders, and a chatbot for providing proactive support to victims/survivors.
2. Promote 'equitable' algorithms which are able to reflect multiple perspectives/viewpoints and not marginalise minority views;
3. Increase digital literacy concerning the safe use of social media from an early age.
In contrast to previous research in this area, our team will include experts on GBV and online harassment to ensure that we frame the problem in a way which is most helpful to victims/survivors of GBV. In other words, we not only focus on *how* to automatically detect online abuse, but also re-think *what* it is we need to detect, how we can *support* the victims and how to *prevent* online GBV through promoting digital citizenship (i.e. prevention and intervention aimed at perpetrators and bystanders).
Our methodology is based on the Scottish Government's Equally Safe strategy and implements a Responsible Innovation Approach in a close collaboration with 3rd sector charities with a long-standing track record in this domain. Our proposal aims to create the following impacts:
1. Create longterm technical solutions to support safer online spaces, including advanced abuse detection tools, tools to automatically generate counter narratives aimed at perpetrators and bystanders, and a chatbot for providing proactive support to victims/survivors.
2. Promote 'equitable' algorithms which are able to reflect multiple perspectives/viewpoints and not marginalise minority views;
3. Increase digital literacy concerning the safe use of social media from an early age.
Publications
Abercrombie G
(2023)
Mirages. On Anthropomorphism in Dialogue Systems
Abercrombie G
(2023)
Mirages. On Anthropomorphism in Dialogue Systems
Bonaldi H
(2024)
NLP for Counterspeech against Hate: A Survey and How-To Guide
| Title | Artist-led community-created drawings mapping online GBV |
| Description | We conducted an artist-led engagement workshop with Ailie Rutherford. Ailie led aseries of pilot workshops with members of the public to practice the artistic form of mapping the perception of GBV into a symbolic language. The artefacts appear in the ESO official project webpage and at the artist's webpage. |
| Type Of Art | Artefact (including digital) |
| Year Produced | 2025 |
| Impact | This was a prototyping activity and we are looking into further funding to take this further to a larger pool of participants including from the Romani ethnic minority in Glasgow. |
| URL | https://sites.google.com/view/equallysafeonline/engagement |
| Description | Over this award we generated new knowledge, datasets and methods/computational models that substantially advance how online gender-based violence (GBV) can be understood, detected, and prevented. Our findings are grounded in large-scale engagement with Non-governmental Organisations (NGOs) that are either frontline case workers of victims/survivors or policymaker and lobbyists. This co-creation approach rooted in feminist research, makes us of the first GBV research programmes in NLP built directly around stakeholder expertise. 1. A new, nationally relevant taxonomy of online GBV We produced the first comprehensive taxonomy of online GBV, drawing on definitions from UN and Equally Safe in Scotland and primarily based on consultation with expert NGOs. This framework has attracted interest from the Scottish Government and provides a much clearer, more structured way of defining the breadth of online GBV behaviours. It challenges existing narrow definitions used in machine-learning research while at the same time provides a clear framework on how to collect and annotate data, and train computational models for detection and mitigation via counterspeech (see objective 3 below). Crucially, this taxonomy has become the foundation for all subsequent datasets and models generated through the project. 2. Creation of major new research resources for the field The award enabled the development of several high-quality datasets that did not previously exist. These include the first conversational online GBV dataset focusing on interactions on Twitter/X including text, images, videos (note that images and videos are just collected; no models were trained on them) and metadata, the first GBV-specific counterspeech dataset, and a large collection of counterspeech strategies informed both by crowd worker annotators and experts. We also produced one of the first in-depth reviews of GBV-related works in NLP, mapping gaps and challenges in the field. In addition, we conducted extensive participatory research with young people through focus groups delivered in partnership with organisations such as YoungScot and Rape Crisis Scotland. Using role-play activities - Tweet Detective (emulating how our detection model identifies harmful content) and Social Media Superhero (mirroring how counterspeech generation models can intervene safely) - young participants explored online harms in an interactive, age-appropriate way. Together, we co-authored a Young People's Report, which was publicly launched at a high-profile event attended by councillors, Scottish Government officials, NGOs and Ofcom. These combined resources enable researchers and third-sector/government stakeholders to explore new questions about victim-support needs, perpetrator behaviour and effective interventions. 3. New machine-learning methods for GBV detection and intervention We developed and evaluated a series of LLM-based models for detecting online GBV, including methods designed to better represent diverse and minority viewpoints often overlooked by mainstream moderation systems. Using part of the developed taxonomy we were able to provide classification models that go beyond binary (exists/doesn't exist) aiming to explain the origins of GBV e.g., in a social media post. The project also explored the use of LLMs and reinforcement learning from human feedback (RLHF) to generate counterspeech aimed at perpetrators or bystanders. These methods improve transparency and interpretability, helping users and practitioners understand why a piece of content is considered harmful. At the same time they offer effective ways to protect and empower victims/survivors without the risk of re-traumatisation by prolonged exposure to toxic content online. 4. Practical tools demonstrating real-world application Although funded separately through an Impact Accelerator Award (IAA), our "Support Buddy" browser extension is a direct offshoot of this research. It demonstrates how the models and resources developed here can support people encountering harmful content online by offering redaction options, content control, reporting pathways, and routes to support services. It provides a tangible model for future practical deployments, and is an avenue that we are actively exploring commercially. Negative or closed-off avenues A key challenge was the lack of high-quality vision-based datasets for online GBV at least not until the very end of the project. Despite initial objectives, we found that available resources were insufficient or impossible to access ethically and legally (e.g., access to TikTok, Instragram is prohibited for open-access research), and obviously developing new ones would pose ethical and safeguarding risks. This clarified the limits of image-based GBV detection and helped redirect the project toward more socially grounded and ethically robust text-based work. Overall, the award delivered new knowledge, methods, datasets, and tools that significantly enhance UK capability to understand and respond to online GBV. |
| Exploitation Route | The outcomes of this award can be taken forward through both academic and non-academic routes. The taxonomy and datasets offer a reusable foundation for researchers in NLP, linguistics, and social sciences to advance survivor-centred detection models and to develop new methodologies for counterspeech, contextual moderation, and digital citizenship interventions. Our findings also highlight ethical and practical limits of vision-based GBV research, calling for resources open to academic research. Beyond academia, third-sector organisations working with victims/survivors can use the taxonomy, counterspeech strategies and model insights to strengthen digital-safety guidance, early-intervention practices, and support services. Policymakers, including those already engaging with our work, can apply the taxonomy to inform regulatory discussions and national online-safety strategies. Tech companies can build on our models to create more transparent, fair, and context-aware moderation tools. The "Support Buddy" browser extension, derived from this research, provides a concrete pathway for adoption by schools, youth organisations, and digital-wellbeing providers, and is a perfect opportunity for commercial exploitation. Taken together, these outcomes equip a wide ecosystem-researchers, NGOs, educators, policymakers, and industry-to advance safer and more supportive online spaces. |
| Sectors | Communities and Social Services/Policy Digital/Communication/Information Technologies (including Software) Government Democracy and Justice |
| Description | Consultation Response to Ofcom (May 2025) |
| Geographic Reach | National |
| Policy Influence Type | Contribution to a national consultation/review |
| URL | https://www.ofcom.org.uk/online-safety/illegal-and-harmful-content/a-safer-life-online-for-women-and... |
| Description | Spanish Government's Department of National Security: 'Conferencia campañas de desinformación y discurso de odio' |
| Geographic Reach | Europe |
| Policy Influence Type | Participation in a guidance/advisory committee |
| URL | https://www.dsn.gob.es/sites/default/files/2025-01/MODIFICADO%20ACCESIBLE%20TRABAJOS%20FORO%20CAMPA%... |
| Description | UK AI Security Institute poll |
| Geographic Reach | National |
| Policy Influence Type | Contribution to a national consultation/review |
| URL | https://www.aisi.gov.uk/work/should-ai-systems-behave-like-people |
| Description | University Research Fund (URF) Impact Acceleration Award for Project 'Support Buddy' |
| Amount | £50,000 (GBP) |
| Organisation | Heriot-Watt University |
| Sector | Academic/University |
| Country | United Kingdom |
| Start | 04/2025 |
| End | 02/2026 |
| Title | GBV-Resources |
| Description | This repository serves as a comprehensive collection of resources for the automated identification of online Gender-Based Violence (GBV) and related phenomena. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2023 |
| Provided To Others? | Yes |
| Impact | Research paper describing the repository available at https://aclanthology.org/2023.woah-1.17/ |
| URL | https://github.com/HWU-NLP/GBV-Resources |
| Title | Re-examining Sexism and Misogyny Classification with Annotator Attitudes |
| Description | Models and data from the research paper Re-examining Sexism and Misogyny Classification with Annotator Attitudes (Jiang et al., Findings of EMNLP 2024) |
| Type Of Material | Computer model/algorithm |
| Year Produced | 2024 |
| Provided To Others? | Yes |
| Impact | The approach taken to modelling annotator attitudes has been followed in several works (e.g. Orlikowski et al., ACL 2025; Khan et al., ACL 2025; Reuver et al., Findings NAACL 2025) |
| URL | https://github.com/HWU-NLP/GBV-attitudes |
| Description | Collaboration with Charities on Gender Based Violence (Y1) |
| Organisation | End Violence Against Women |
| Country | United Kingdom |
| Sector | Charity/Non Profit |
| PI Contribution | The 3 main charities (End Violence Against Women (EVAW), The Glitch, and EmilyTest) engage in supporting victims of Gender-based Violence, campaigning, and policymaking at the local government (Scotland) and UK level. One of the named researchers on the project (Poppy Gerrard-Abbott) has been previously working for EmilyTest and is leading the effort of coordinating workshops and focus groups with the partners in particular to co-create an online GBV taxonomy. |
| Collaborator Contribution | The involvement of the 3 main charities is a key component of the project, which requires the contribution of experts in GBV (rather than Computer Science experts) to help co-create the online GBV taxonomy and dataset annotations. |
| Impact | - 1 half-day workshop with an emphasis on giving feedback on the first draft of the online GBV taxonomy (see also the relevant outcome on the portfolio). |
| Start Year | 2022 |
| Description | 1st Workshop on online GBV Taxonomy |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | 1st workshop (half-day; September 2023) with the experts from the 3 main charities only (End Violence Against Women (EVAW), The Glitch, and EmilyTest) that engage in supporting victims of Gender-based Violence, campaign and policymaking at the local government (Scotland) and UK level. The main objectives were to receive feedback on the first draft of the online GBV taxonomy that the award-holder institute (Heriot-Watt) and main partner (University of Edinburgh) had been working on during Y1 of the project. The feedback we received influenced the next steps: a) Change the focus of the online GBV taxonomy we had established thus far to the respective parts of "textual expressions" and "cultural microaggressions" of the pyramid of discrimination and violence, rather than the extremes of it, i.e., "physical expression" and attitudes and beliefs. b) (ongoing as of the time of submission of this outcome) Conduct several small focus groups of experts beyond the 3 main charity partners to help validate the framework. The focus groups comprise mostly experts "on the street" that directly support victims of GBV such as rape, or bereaved families. b) identify the appropriate online venues for sourcing data of online GBV for detection annotation, c) conduct initial discussions (only applicable from Y2 and on) on how to further recruit GBV experts from the charities for judging existing counter speech strategies. |
| Year(s) Of Engagement Activity | 2023 |
| Description | 2nd Workshop on Perspectivist Approaches to NLP (NLPerspectives 2023) |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Other audiences |
| Results and Impact | This full-day workshop was held at the European Conference on AI (ECAI) in Krakow, Poland on the 30th September 2023. It was attended by 42 researchers and featured a keynote and panel featuring international guests. Nine research papers were presented including five archival papers published in the workshop proceedings at https://ceur-ws.org/Vol-3494/ |
| Year(s) Of Engagement Activity | 2023 |
| URL | https://nlperspectives.di.unito.it/w/2nd-workshop-on-perspectivist-approaches-to-nlp/ |
| Description | Adult Focus Groups with NGOs to formulate oGBV taxonomy |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Third sector organisations |
| Results and Impact | We conducted a series of focus groups with a total of 11 NGOs that specialise in Gender Based Violence either as front-line workers or policymaking/campaigning (Edinburgh Rape Crisis, Rape Crisis National, SafeLives, Scottish Women Convention, Revenge Porn Helpline, Refuge?, Our Streets Now, EmilyTest (core partner), End Violence Against Women (core partner), Glitch (core partner), AMINA, Scottish Women's Aid). Each focus group activity involved a few members of each organisation who conducted a couple of exercises to inform the creation of the online GBV taxonomy. This was a vital part of one of our key objectives successfully achieved; the taxonomy is completed and is currently informing a large annotation effort. It will also play a key role in updating the 'Equally Safe Act Strategy' in Scotland which is due to take place by the end of 2025. Several participants also reported benefiting from interacting with our researchers and engaging in the co-creation of the taxonomy. |
| Year(s) Of Engagement Activity | 2024 |
| Description | Empowering women using A.I - TEDxYouth@StGeorgesEdinburgh |
| Form Of Engagement Activity | A broadcast e.g. TV/radio/film/podcast (other than news/press) |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Public/other audiences |
| Results and Impact | I gave a TEDx Youth talk at St George's School in Edinburgh in March 2024 that appeared on the official TED Youtube channel later in May. The talk was on the role of AI in empowering women through the detection and mitigation of GBV online and it was addressed to the general public. Although there was no QA session with the audience (the event was designed for recording purposes only), the conversations that followed with participants showed great interest and increase in awareness. It was advertised on Linkedin and through the project's website as well. |
| Year(s) Of Engagement Activity | 2024 |
| URL | https://www.youtube.com/watch?v=h1o1ABrWDyA |
| Description | Invited Talk at Archimedes Summer School 2023 |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Postgraduate students |
| Results and Impact | Invited talk on risks of Generative AI for Question Answering with respect to faithfulness and trust. The audience was mostly early-year PhD students in Greece that were taking part on the 2-day event of the summer school. The talk created a lot of debate given the controversial nature of Generative AI and potential harms, and resulted in follow-up meetings with two students. |
| Year(s) Of Engagement Activity | 2023 |
| URL | https://archimedesai.gr/en/two-day-workshop-on-natural-language-processing |
| Description | Invited talk at Google Deep Mind, London |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Industry/Business |
| Results and Impact | An invited talk at the Sociotechnical AI Research group, Google Deep Mind. Led to a lively discussion on fairness and ethics in NLP with industry practitioners. |
| Year(s) Of Engagement Activity | 2023 |
| Description | Invited talk at Napier University School of Computing |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Other audiences |
| Results and Impact | An invited talk for academics and postgradute students working on NLP. This led to a paper on participatory design in NLP which we are currently working on in collaboration with Napier University. |
| Year(s) Of Engagement Activity | 2023 |
| Description | Invited talk at the National Robotarium |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Other audiences |
| Results and Impact | An invited talk presenting recent work related to the Equally Safe Online and Gender Bias in Conversational Agents projects. Attended by members of faculty, students and members of the public. |
| Year(s) Of Engagement Activity | 2023 |
| Description | Kickoff meeting with Core Charity partners |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | Kickoff meeting of the project with all the stakeholders including the 3 main charities (End Violence Against Women (EVAW), The Glitch, and EmilyTest) that engage in supporting victims of Gender-based Violence, campaign and policymaking at the local government (Scotland) and UK level. The main outcome was to receive guidance and steer the award-holder institute (Heriot-Watt) and main academic partner (University of Edinburgh) toward the primary goals of the project: a) establish the correct methodology for creating the Online GBV framework, b) identify the appropriate online venues for sourcing data of online GBV for detection annotation, c) conduct initial discussions (only applicable from Y2 and on) on how to further recruit GBV experts from the charities for judging existing counter speech strategies. |
| Year(s) Of Engagement Activity | 2023 |
| Description | Launch Event of Young People's report |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Policymakers/politicians |
| Results and Impact | Young People versus Gender Based Violence' - Launch event a tthe Informatics Forum, University of Edinburgh. This was a high-profile invitation-only event in collaboration with our partners at UoE to mark the release of Young People report on GBV. We attracted ~80 participants including government officials, MSPs, councillors, Police Scotland and NGOs. The follow-up conversations will lead to potential funded projects/customers for the 'Support Buddy' spin-out (see Narrative Impact for more details). |
| Year(s) Of Engagement Activity | 2025 |
| Description | Presentation to Greek Ambassador in the UK at the National Robotarium |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Policymakers/politicians |
| Results and Impact | Presented an overview of the key objectives and outcomes of the project to a delegation from the Greek Embassy in the UK including the ambassador Ioannis Tsaoussis and a member from the research commercialisation team. The discussion afterwards revolved around impact-oriented practical applications for detecting GBV online that could be supported by joint Greek-UK funding. The event was posted on Linkedin afterwards and attracted some attention. |
| Year(s) Of Engagement Activity | 2024 |
| URL | https://www.linkedin.com/posts/mariakoskinopoulou_touchlab-visionrf-robotics-activity-72569373102298... |
| Description | The 1st Workshop on CounterSpeech for Online Harms (CS4OA) |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Other audiences |
| Results and Impact | The workshop was collocated with the 24th meeting of SIGDial (Special Interest Group on Discourse and Dialogue), and brought together researchers form computer science and experts in policy surrounding tackling online abuse and hate speech. Eight research papers were presented along with two invited keynote talks and a panel discussion. |
| Year(s) Of Engagement Activity | 2023 |
| URL | https://sites.google.com/view/cs4oa |
| Description | Young Persons Focus Groups |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Public/other audiences |
| Results and Impact | We conducted a series of half-day focus groups with young participants (13+) in collaboration with 3 NGOs (Rape Crisis Ayrshire, Rape Crisis Shetland, Rape Crisis Grampian, YoungScot). The idea behind the workshops was to ask the opinion of a younger audience about GBV online drawing from their experience as users and in several cases as victims/survivors. The data collected validated the taxonomy that was co-created earlier via the adult focus groups (see relevant activity) and also informed the strategies we implemented in the Generative AI models we developed for Counter speech. All participants felt empowered by the opportunity to inform the process of creating models to address GBV. |
| Year(s) Of Engagement Activity | 2024,2025 |