Streamlining Social Decision Making for Improved Internet Standards

Lead Research Organisation: Queen Mary University of London
Department Name: Sch of Electronic Eng & Computer Science

Abstract

Many decisions in today's world are made through a complex, dynamic process of interaction and communication between people and teams with different interests and priorities - so called "distributed decision-making" (DDM). For example, many businesses work across multiple geographically dispersed offices and timezones, with teams specialising in quite diverse areas. Each team may have its own goals and reward models, which do not necessarily coincide, and may be spread across multiple organisational units (e.g. different businesses or governments). Communication may happen via several different modalities with very different timescales and properties (e.g. email, instant messenger, and face-to-face meetings).

Unfortunately, although many organisations have started to document these processes and even make records available (particularly governmental organisations e.g. https://data.gov.uk/), we have no way to automatically analyse these records. If we did, we could produce tools to automatically summarise decisions, trace who made them, and why and how they were made (and why other decisions weren't made). From a societal standpoint this would help make these processes more accountable and transparent. We'd also be able to identify collaborative failures, biases and other problems, and thus help improve decision-making in future.

This project will develop these urgently required methods, using a combination of natural language processing and social network analysis. We will collate, annotate and publicly release the first multimodal dataset of real-world distributed decision-making. We will devise techniques to take natural language and semi-structured data to recognise the dialogue and interaction structures in decision making, and analyse those structures to produce summaries and evaluate the efficacy of the decision making process. We will then use the outputs to inform strategic interventions that can streamline and improve decision making.

Our methods will be suitably generic to span several domains. However, the project will focus on one particular global organisation as its main use case: the Internet Engineering Task Force (IETF). This is an international forum responsible for producing Internet protocol standards - formal documents which specify the languages by which software and hardware "speak" across the Internet. To produce these documents, extensive international collaboration is performed - this spans several modalities including email discussions, collaborative document editing, face-to-face meetings and teleconferencing. Importantly, all of these modalities are documented via transparency reports ranging from public email archives to minutes from meetings. This project has partnered with the IETF to help model and streamline their decision making process. We will borrow from their experience, and employ our methods to extract decision making bottlenecks. We will devise tooling which will provide advice and proposed interventions to relevant parties within the IETF. Amongst many other things, we directly benefit the IETF, and the global Internet standards community, by helping them to uncover biases and help make important decision processes accountable.

Planned Impact

The project will generate ground-breaking advances in the automated processing of natural language and semi-structured data relating to social decision making. By partnering with the IETF, it will also apply these results to improve the process of developing Internet standards, and to streamline the IETF's decision making process. We envisage impacts from both of these sets of outcomes, and their combination across various themes, discussed below:

Novel research approaches: By bringing together researchers in NLP, social network analysis, computer networks and psychology we expect novel synergy-producing approaches with impact across academic disciplines (see Academic Beneficiaries).

Datasets and tools: By producing and publicly releasing the first large-scale multimodal dataset of real-world distributed decision-making data we will benefit researchers in many areas (see Academic Beneficiaries). We will ensure that all publications, documentation, data and software are made publicly available e.g. via GitHub and the UK Data Archive to encourage uptake and re-use.

Technology transfer: The new methods we will develop will lead to economic and societal impact via technology transfer. Methods for understanding multi-modal human-human interaction (not only in decision-making but also in many social and business environments) will be commercially valuable for building dialogue systems, e.g. chatbots, business and decision support systems (amongst other possibilities). Further, our impact plan will help transfer this to commercial reality via spinout formation and/or licensing to existing UK industry including companies working with our partner organisation, the IETF. This will be supported by QMUL's technology transfer company, QM Innovation Ltd and dedicated mobile app division QApps (www.qappsonline.com); and/or dedicated funding via InnovateUK or similar. The PI has a strong track record of spinout formation, licensing and InnovateUK transfer from NLP research (Chatterbox Labs, IESO Digital Health, Quality Health Ltd).

Impact on the IETF: Our key impact partner is the IETF itself, which will deploy our tools for streamlining and improving their activities. Our research agenda has been coordinated with direct inputs from the IETF, such that the impact can be maximised. Our key long-term impact will come from the improvements to the IETF procedures derived from the project's interventions. Other impacts will be to improve the quality of standards, by better supporting cross-area review, and to help improve community representation by highlighting bottlenecks and biases in the process. To further the impact and extend it beyond the life of this project, we will work towards the integration of our tools with the IETF's datatracker (a web portal, which exposes public data about standards activities).

Impact on industry and society via internet standards: this positive impact on internet standards, and particularly opening them up to inputs from more diverse groups, will feed into better-met industry needs. Since this will have a direct benefit on the technical decisions underpinning how the Internet operates, the impact will eventually extend to most members of society. The UK is a key player in IETF, and we will work with industry to demonstrate how our techniques can improve their effectiveness, ensuring future Internet standards continue to reflect UK interests. Our impact plan therefore focuses on reaching and influencing three main constituencies: (1) organisations/companies interested in standardisation and social decision making (Sky, Ericsson, JISC); (2) the IETF, which has an inherent interest in improving their activities (Eggert, Navarro, Ford, Oever); and (3) the research community interested in exploring social decision making in other domains, as well as those interested in the standardisation process and the interplay of the agents building the Internet (Lascarides, Doty).
 
Description Analysis of the communication patterns in the Internet Engineering Task Force (IETF) have revealed several findings about communication and decision-making in large distributed organisations. For example, we have found that participants' language is linked to their levels of influence; that leaders use distinctive language, avoiding terms that might be construed as sensitive even when not using them literally; that communication patterns can reveal how responsive people are at different levels of a hierarchy; and that communication patterns and people's roles in them change in characteristic ways during the decision-making process. We have developed tools to detect many of these aspects automatically.

These findings have led to the creation in 2023 of a new Research Group at the Internet Research Task Force (IRTF), which now advises the IETF on best practices and potential improvements to the standardisation processes (see Narrative Impact); and to ongoing collaboration with the IETF developing software tools to predict the time required for standards documents to be edited and agreed.
Exploitation Route The findings on decision-making analysis can be applied to a broad range of organisations to help streamline communication and decision-making. We have already begun work within the Internet Research Task Force (IRTF), setting up a new research group that now advises the IETF on best practices and potential improvements to the standardisation processes (see Narrative Impact); and work with the IETF developing software tools to predict the time required for standards documents to be edited and agreed.
Sectors Digital/Communication/Information Technologies (including Software)

Financial Services

and Management Consultancy

URL https://sodestream.github.io/
 
Description Project results and their dissemination at talks led to a new Research Group (RG) at the Internet Research Task Force (IRTF), the Research and Analysis of Standardisation Processes Research Group (RASP RG). The new RASP group influences the practice of internet standards development by bringing together policy makers, researchers, practitioners and industry representatives to analyse and identify best practices and potential improvements to the standardisation processes in the Internet Engineering Task Force (IETF) and other Standard Development Organisations (SDOs). The group has so far met four times, with 65-90 people in attendance, and will meet again at the next IETF meeting in March 2024. Open standardisation forums are fundamental in digital technologies and the Internet. The IETF has developed and maintain most of the protocols that underpin the Internet since its inception up to today. The open nature of this process creates shortcomings and inefficiencies that RASP analyses and reflects in appropriate documents and presentations at IETF meetings, informing the IETF leadership of potential improvements. RASP RG charter: http://datatracker.ietf.org/doc/charter-irtf-rasprg/
First Year Of Impact 2023
Sector Digital/Communication/Information Technologies (including Software)
Impact Types Societal

Economic

 
Description Creation of the Research and Analysis of Standardisation Processes Research Group (RASP RG) at the Internet Research Task Force (IRTF)
Geographic Reach Multiple continents/international 
Policy Influence Type Influenced training of practitioners or researchers
Impact RASP influences the practice of standards development by bringing together policy makers, researchers, practitioners and industry representatives to analyse and identify best practices and potential improvements to the process. Open standardisation forums are fundamental in digital technologies and the Internet. The IETF has developed and maintain most of the protocols that underpin the Internet since its inception up to today. The open nature of this process creates shortcomings and inefficiencies that RASP analyses and reflects in appropriate documents and presentations at IETF meetings.
URL http://datatracker.ietf.org/doc/charter-irtf-rasprg/
 
Description IETF project to streamline decision making in the edition of documents
Geographic Reach Multiple continents/international 
Policy Influence Type Contribution to new or improved professional practice
Impact Once in production, this will help to better allocate the resource of the IETF.
 
Title Datasets and analysis scripts for IETF social graph analysis, to support ICWSM 2022 paper 
Description The Internet Engineering Task Force (IETF) has developed many of the technical standards that underpin the Internet. The standards development process followed by the IETF is open and consensus-driven, but is inherently both a social and political activity, and latent influential structures might ex- ist within the community. Exploring and understanding these structures is essential to ensuring the IETF's resilience and openness. We use network analysis to explore the social graph of IETF participants, based on public email discussions and co-author relationships, and the influence of key contributors. We show that a small core of participants dominates: the top 10% contribute almost half (43.75%) of the emails and come from a relatively small group of organisations. On the other hand, we also find that influence has become relatively more decentralised with time. IETF participants also propose and work on drafts that are either adopted by a working group for further refinement or get rejected at an early stage. Using the social graph features combined with email text features, we perform regression analysis to understand the effect of user influence on the success of new work being adopted by the IETF. Our findings shed useful insights into the behavior of participants across time, correlation between influence and success in draft adoption, and the significance of affiliated organisations in the authorship of drafts. 
Type Of Material Data analysis technique 
Year Produced 2022 
Provided To Others? Yes  
Impact No specific impact tracked so far 
URL https://github.com/sodestream/icwsm22
 
Title IETF 115 Hackathon datasets 
Description The ietfdata library provides a Python 3 interface to access the IETF Datatracker, RFC index, and mailing list archives. The library can also make use of a MongoDB instance as a cache, reducing the number of requests that are made directly to the Datatracker, improving performance, and reducing impact on the IETF's infrastructure. The library's README provides further setup information 
Type Of Material Database/Collection of data 
Year Produced 2022 
Provided To Others? Yes  
Impact Used by c.10 IETF research teams at IETF 115 
URL https://github.com/sodestream/ietf115-hackathon
 
Title LEDA: a Large-Organization Email-Based Decision-Dialogue-Act Analysis Dataset 
Description Dataset based on real-world data from the public mail-archives of the Internet Engineering Task Force (IETF), annotated with a set of dialog acts for the study of decision-making mechanisms in large distributed groups. 
Type Of Material Database/Collection of data 
Year Produced 2023 
Provided To Others? Yes  
Impact Paper published in Findings of the ACL 2023. 
URL https://github.com/sodestream/acl2023-email-da-dataset
 
Title Tracing Linguistic Markers of Influence in a Large Online Organisation 
Description Dataset from email archives of the Internet Engineering Task Force (IETF), supporting analysis of linguistic differences associated with participants' levels of influence. 
Type Of Material Database/Collection of data 
Year Produced 2023 
Provided To Others? Yes  
Impact Paper published at ACL 2023 
URL https://github.com/sodestream/acl2023-tracing-linguistic-markers
 
Description Toshiba Cambridge Research Laboratory 
Organisation Toshiba Research Europe Ltd
Department Cambridge Research Laboratory - Toshiba
Country United Kingdom 
Sector Private 
PI Contribution Meetings for research discussion and grant preparation
Collaborator Contribution Meetings for research discussion and grant preparation
Impact Dissemination of research results, and new collaborative grant proposal preparation in progress.
Start Year 2021
 
Description University of Glasgow 
Organisation University of Glasgow
Department School of Computing Science
Country United Kingdom 
Sector Academic/University 
PI Contribution Explicit collaboration as academic partners on this award.
Collaborator Contribution Explicit collaboration as academic partners on this award.
Impact Joint publications in progress.
Start Year 2020
 
Title Code to support "Errare humanum est: What do RFC Errata say about Internet Standards?" 
Description Code to support "Errare humanum est: What do RFC Errata say about Internet Standards?", published in the proceedings of the 7th Network Traffic Measurement and Analysis Conference (TMA 2023) --https://eprints.gla.ac.uk/298487/ 
Type Of Technology Software 
Year Produced 2023 
Impact Paper published at TMA 2023 
URL https://zenodo.org/record/8008031
 
Title Software to support analysis of email dialogue act modelling and prediction 
Description Software to support analysis of email dialogue act modelling and prediction 
Type Of Technology Software 
Year Produced 2023 
Open Source License? Yes  
Impact Paper published at Findings of the ACL 2023 
URL https://github.com/sodestream/acl2023-email-da-dataset
 
Title Software to support linguistic and influence analysis in email 
Description Software to support linguistic and influence analysis in email 
Type Of Technology Software 
Year Produced 2023 
Open Source License? Yes  
Impact Paper published at ACL 2023 
URL https://github.com/sodestream/acl2023-tracing-linguistic-markers
 
Description An Email Dataset for Analyzing Large-Group Decision-Making 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact The talk presented a novel dataset and how can it be used by other parties. This sparked questions, potential future collaborations and general interest in using this dataset
Year(s) Of Engagement Activity 2023
URL https://sites.google.com/view/icwsm2023datachallenge/home
 
Description Exploring Mis(Use) on Messaging Platforms 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact This was a research presentation and visit at the Asian Institute of Technology in Bangkok. It accompanied a research visit to the institute to perform knowledge exchange. This involved visiting Prof. Kanchana Kanchanasut, a member of the Internet Hall of Fame, and a winner of the Jonathan B. Postel Service Award. Prof. Kanchanasut is heavily involved in the IETF, bringing their physical meeting to Bangkok once before and currently bringing it again (date to be confirmed due to COVID-19). The purpose of the visit was to learn about IETF activities in Thailand and other similar regions, as well as disseminate information about the project.
Year(s) Of Engagement Activity 2022
 
Description Internet Global Forum presentation 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Policymakers/politicians
Results and Impact Presentation at the IGF which resulted in an interesting debate with the audience on the relevance of the topic and how to bridge the gap of understanding across the multiple stakeholders
Year(s) Of Engagement Activity 2023
URL https://dig.watch/?post_type=event&p=149752
 
Description Keynote speaker at AINTEC 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Postgraduate students
Results and Impact I was invited as a keynote speaker to present my research and findings at the 17th Asian Internet Engineering Conference (AINTEC) to an audience of academics, PhD students and industry practitioners in Hiroshima University, Japan. This resulted in a very lively debate and lengthy discussion afterwards with multiple follow ups on the findings of the research and its future directions.
Year(s) Of Engagement Activity 2022
URL http://www.interlab.ait.ac.th/aintec2022/
 
Description New Research Group at the Internet Research Task Force 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact I created and chair a Research Group at the Internet Research Task Force. The Research Group meets 3 times a year at associated Internet Engineering Task Force Meetings. In the multiple meeting of the research group, there has been a wide discussion and interest in follow up activities, and engagement.
Year(s) Of Engagement Activity 2023,2024
URL https://datatracker.ietf.org/rg/rasprg/about/
 
Description Presentation at the Measurement and Analysis for Protocols Research Group (maprg) 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact About 100 people attended the maprg session. Our research sparked questions and discussions and there was significant interest that later resulted in the creation of an official Research Group dedicated to further the analysis of standard development organisation (eg, IETF).
Year(s) Of Engagement Activity 2022
URL http://datatracker.ietf.org/meeting/115/materials/agenda-115-maprg-12
 
Description Presentation to Centre for Language and Studies in Probability (CLASP) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact Presentation of research questions and methods being used on the EMBEDDIA and SoDeStream projects. 10-15 attendees from CLASP at the University of Gothenburg, Sweden.
Year(s) Of Engagement Activity 2020
 
Description Presentation to Toshiba Research Lab 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Industry/Business
Results and Impact Presentation of the research questions and methods being used on the SoDeStream project, focusing on modelling decision-making processes and interaction. 10-15 attendees from Toshiba's research labs in the UK and Japan.
Year(s) Of Engagement Activity 2021
 
Description Presentation to Toshiba Research Lab 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Industry/Business
Results and Impact Presentation to Toshiba Research Lab
Year(s) Of Engagement Activity 2024
 
Description Presentation to the Internet Internet Engineering Steering Group (IESG) 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact The Internet Engineering Steering Group (IESG) is a key group in the IETF who is responsible for the technical management of IETF activities and the Internet standards process. At the request of the IESG we presented and discussed our research and preliminary findings.
Year(s) Of Engagement Activity 2020
URL https://csperkins.org/research/protocol-standards/2020-12-10-ignacio-iesg-talk/
 
Description Presentation to the Internet Internet Engineering Steering Group (IESG) 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Presentation and discussion with the Internet Engineering Steering Group (IESG). IESG is a key group in the IETF who is responsible for the technical management of IETF activities and the Internet standards process. We presented our research and findings and discussed the bottlenecks and potential improvements that open standardisation processes and the IETF in particular have. Note that this meeting is restricted to the IETF leadership and therefore does not make its agenda and minutes public.
Year(s) Of Engagement Activity 2022
URL http://datatracker.ietf.org/meeting/115/
 
Description Presentation to the Working Group Chairs Forum of the IETF 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact The findings of our research were presented to the chairs of the IETF working groups. We identified shortcoming and potential improvements and debated these with the participants. The significant interest in our work later resulted in the creation of an official Research Group dedicated to further the analysis of standard development organisation (eg, IETF). Note that this meeting is restricted to working group chairs and therefore does not make its agenda and minutes public.
Year(s) Of Engagement Activity 2022
URL http://datatracker.ietf.org/meeting/115/proceedings
 
Description Survey of IETF reviewer matching preferences 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Survey to gather views and preferences on automatic reviewer matching support, to feed into research experiments and tool development
Year(s) Of Engagement Activity 2022,2023
URL https://ietf-annotation.eecs.qmul.ac.uk/html/index.html
 
Description Talk at RIPE 87 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact The RIPE 87 meeting had 556 physical attendants and 455 remote ones. The presentation sparked a debate with practitioners and attendants engaged in policy making.
Year(s) Of Engagement Activity 2023
URL https://ripe87.ripe.net/archives/video/1247/
 
Description Talk at a IAAA ICWSM workshop (Flocking to Mastodon: Tracking the Great Twitter Migration) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact The talk discussed how users transitioned from Twitter to Mastodon and how their language evolved and whether conversations were more or less toxic in one platform or the other
Year(s) Of Engagement Activity 2023
URL https://sites.google.com/view/icwsm2023datachallenge/home