Twenty20Insight

Lead Research Organisation: Aston University
Department Name: College of Engineering and Physical Sci

Abstract

As digital technology permeates every area of modern life, we risk becoming over-dependent on complex systems that operate in an opaque way, creating a risk that they exhibit emergent properties that adversely affect their users or their wider environment. This is particularly true as developers increasingly rely on AI or ML techniques as a means to define system behaviour when the problem space is too complex or poorly understood for human developers to explicitly specify that behaviour. We are tackling incompletely understood problems by developing systems whose behaviour and wider impact are by necessity also incompletely understood. This trend, which is largely enabled by an abundance of data harvested from (e.g.) mobile devices, sensors and social media, is radically changing how systems are developed and how they are used. We need a new approach to software engineering that

(i) places greater emphasis on making explicit the risks of unintended behaviour for innovative new software products either through limitations on our understanding of the envisioned product's behaviour or through misuse, and

(ii) actively supports explainability of the exposed behaviour by the running system.

Twenty20Insight is an interdisciplinary project bringing together academic experts in Software Engineering (SE), RE, Design Thinking and ML to help system stakeholders and developers understand and reason about the impact of intelligent systems on the world in which they operate.

Planned Impact

We expect the project to have a major impact on five classes of beneficiaries beyond the academic research community:

Industrial partners. Twenty20Insight will have an early impact on our industrial partners, with whom we will work to advance the state-of-the-art in stakeholder engagement for the co-design of intelligent systems, and techniques for reasoning about uncertainty and making ML systems' decision-making more transparent.


Early career researchers. The project will provide the attached PDRAs with advanced training in research methods, publication and career development, which will eventually develop the PDRAs into independent researchers by the end of the project.

Research community. Twenty20Insight will develop fundamental new knowledge about engineering intelligent systems both within and at the boundaries between the design, RE/SE, AI and ML research communities.

Industries revolutionised by AI. AI/ML can be applied to nearly every industrial sector. Designing the increasingly complex AI systems requires stakeholders to understand both the Horizon of Possibilities and Envelope of Acceptability when applying ML to their problems. This project has a great potential to tackle such a problem and thus will significantly impact all the industries revolutionised by AI.

Software development companies. The new tools and techniques that will be developed in this project will have the potential to enhance software development companies to meet the increasing demand for intelligent systems operating in an uncertain world.

Wider society. The principal beneficiary of initiatives that aim to generate transformative ideas in order to advance technology, such as the Twenty20Insight project, is ultimately the wider society. This is particularly true of Twenty20Insight since one our explicit aims is to discover techniques to better understand systems' societal impact and enable stakeholders to handle such knowledge in a systematic and transparent way.
 
Description In Twenty20Insight, we are tackling the need for software engineering techniques that:

(i) places greater emphasis on making explicit the risks of unintended behaviour for innovative new software products either through limitations on our understanding of the envisioned product's behaviour or through misuse, and
(ii) actively supports explainability of the exposed behaviour by the running system.

Let's remember the objectives to describe to what extent have the award objectives met so far:

O1: stakeholders are able to explore how envisioned systems might interact with and impact on the external world in which they will operate - to explore the horizon of possibilities (HoP) and define the Envelop of Acceptability (EoA); [WP1,WP2]

O2: developers can model and reason about the uncertainty that is an inevitable residual feature of the system design, even after the definition of the EoA; [WP1,WP2 and WP3]

O3: advance the extent to which ML algorithms behave can be rendered interpretable by humans [W3 and WP4, WP5]

Twenty20Insight started in September 2020 (in the middle of the pandemic). Since then, even with the challenges posed by the pandemic, we have been able to work of different collaborations and publications towards the 3 goals.

We have the following achievements and findings:

a. Techniques for the quantification of the impact of values on requirements and software engineering (O1, WP2, WP3). 2 Conference Publications and a workshop publications.

b. The technique and software architecture (Pri-AwaRE) to create runtime awareness about priorities of quality properties (i.e. how much value is given to a property) and measure degrees of uncertainty to support autonomous decision-making (O2, O3, WP3, WP4, WP5) 1 journal publication, 2 conference publication, 1 Exemplar.

c. Techniques (ETeMoX, Cronista) for the explanation of autonomous systems based on ML/AI (O2, O3, WP3,WP4). 2 journal publications, 4 conference publications

d. First steps towards incorporating speculative techniques in Requirements Engineering to better anticipate unwanted emergent behaviour in uncertain environments. (O1, WP1, WP2). 1 Conference paper under submission, 2 workshops delivered to the industry partners. 2 workshops in the next two weeks.
Exploitation Route Based on our collaborations and product with the Industry partners
In the international conferences REFSQ and RE (we have created a workshop to run, which was accepted)
Sectors Digital/Communication/Information Technologies (including Software)

URL https://cs.aston.ac.uk/Twenty20Insight/
 
Description We have taken the first steps towards incorporating speculative techniques in Requirements Engineering to better anticipate unwanted emergent behaviour in uncertain environments. The process has been used in the delivery of a couple of workshops with our Industry partners. We hope to have concrete results in the second stage of the project.
First Year Of Impact 2022
Sector Digital/Communication/Information Technologies (including Software)
Impact Types Cultural,Societal,Economic

 
Title RDMSim: An Exemplar for Evaluation and Comparison of Decision-Making Techniques for Self-Adaptation 
Description To enable researchers to evaluate and compare decision-making techniques for self-adaptation, we present the RDMSim exemplar. RDMSim enables researchers to evaluate and compare techniques for decision-making under environmental uncertainty that support self-adaptation. The focus of the exemplar is on the domain problem related to Remote Data Mirroring, which gives the opportunity to face the challenges described above. RDMSim provides probe and effector components for easy integration with external adaptation managers, which are associated with decision-making techniques and based on the MAPE-K loop. Specifically, (i) RDMSim, a simulator for real-world experimentation, (ii) a set of realistic simulation scenarios that can be used for experimentation and comparison purposes, (iii) data for the sake of comparison. 
Type Of Material Database/Collection of data 
Year Produced 2021 
Provided To Others? Yes  
Impact The RDMSim exemplar presented is implemented in Java, keeping in view the operational model presented in [1], [2]. It simulates the RDM presenting a fully connected network of mirrors. The simulator offers the flexibility of changing the number of mirrors to create a customized RDM network according to the experiment's requirements. Additionally, we provide an implementation of different scenarios that define possible different uncertain environmental contexts for the RDM [20]. A Python version is also publicly available. Researchers can use these scenarios to test their specific decision-making techniques based on Reinforcement Learning [21], Multi-Criteria Decision Analysis [22] and Evolutionary Computation [3] among others. Researchers can also design their own scenarios by modifying the different parameter ranges. [1] M. Ji, A. C. Veitch, J. Wilkes et al., "Seneca: remote mirroring done write." in USENIX Annual Technical Conference, General Track, 2003, pp. 253-268. [2] K. Keeton, C. Santos, D. Beyer, and J. Chase J.and Wilkes, "Designing for disasters," USENIX Conference on File and Storage Technologies, Berkeley, 2004. [20] N. Esfahani, E. Kouroshfar, and S. Malek, "Taming uncertainty in self-adaptive software," in Proceedings of the 19th ACM SIGSOFT symposium and the 13th European Conference on Foundations of software engineering - SIGSOFT/FSE '11. Szeged, Hungary: ACM Press, 2011, p. 234. [21] H. Samin, L. Garcia Paucar, B. Nelly, and P. Sawyer, "Towards priority awareness in autonomous intelligent systems," in 36th ACM/SIGAPP Symposium On Applied Computing (SAC). ACM, 2021. [22] E. Triantaphyllou, "Multi-criteria decision-making methods," in Multicriteria decision making methods: A comparative study. Springer, 2000, pp. 5-21. 
URL https://zenodo.org/record/4613152#.YjJkHZPP0qw
 
Description Invitation to talk at CS Department, Birmingham University - Seminars of CS Department (March 2020) - Online due to Pandemic 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Postgraduate students
Results and Impact Invitation to talk at CS Department, Birmingham University - Seminars of CS Department (March 2021)

Seminar for Academics and Postgraduate Students
Year(s) Of Engagement Activity 2021
 
Description Invitation to talk at CS Department, Durham University - Seminars of CS Department (April 2020) - Online due to Pandemic 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Professional Practitioners
Results and Impact Invitation to talk at CS Department, Durham University - Seminars of CS Department (April 2021)
Year(s) Of Engagement Activity 2021
 
Description Invitation to talk at CS Department, Leeds University - Seminars of CS Department (February 2022) - Online due to Pandemic 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Postgraduate students
Results and Impact The Seminar was for a cohort of 100 postgraduate students (talk online).
Year(s) Of Engagement Activity 2022
 
Description Workshop on Design Thinking for Arcadis 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Professional Practitioners
Results and Impact Arcadis Airport Design Futures workshop

1. Longer-term horizon 2035

Inputs are background research on future air travel visions, technology development, passenger attitude and experience surveys.
The outputs were illustrated scenarios of future air travel / airport experience focus, and a report on implications for investment decisions, cross referenced to a range of alternative scenarios.
Participants: ARCADIS clients (airport managers, executives), other interested parties- Airlines, travel/holiday vendors, UK Government- civil servants.

Aims:
(i) to explore the horizon of possibilities in future air travel and develop a range of visions to illustrate possible futures as videos, VR, MURAL demos, etc
(ii) to develop evaluation criteria and use these to critique future visions to inform policy of passenger experience/marketing and investment decisions in airport facilities
Year(s) Of Engagement Activity 2021