PLEAD: Provenance-driven and Legally-grounded Explanations for Automated Decisions

Algorithms and Artificial Intelligence play a key role nowadays in many technological systems that control or affect various aspects of our lives. They optimise our driving routes every day according to traffic conditions; they decide whether our mortgage applications get approved; they even recommend us potential life partners. They work silently behind the scene without much of our notice, until they do not. Few of us would probably think much about it when our credit card application is approved in two seconds. Only when it is rejected, do we start to question the decision. Most of the time, the answers we get are not satisfactory, if we get any at all. The spread of such opaque automated decision-making in daily life has been driving the public demand for algorithmic accountability - the obligation to explain and justify automated decisions. The main concern is that it is not right for those algorithms, effectively black boxes, to take in our data and to make decisions affecting us in ways we do not understand. For this reason, the General Data Protection Regulation requires that we, as data subjects, be provided with "meaningful information about the logic involved, as well as the significance and the envisaged consequences of such processing." Likewise, consumers should be treated fairly when receiving financial services as per financial services regulations and algorithms should be free of discrimination as per data protection, equality and human rights laws. However, as laws and regulations do not prescribe how to meet such requirements, businesses are left with having to interpret those themselves, employing a variety of means, including reports, interactive websites, or even dedicated call centres, to provide explanations to their customers.

Against this background, provenance, and specifically its standard PROV, describes how a piece of information or data was created and what influenced its production. Within recorded provenance trails, we can retrace automated decisions to provide answers to some questions, such as what data were used to support a decision, who or what organisation was responsible for the data, who else might have been impacted. While provenance information is structurally simple, provenance captured from automated systems, however, tends to be overwhelming for human consumption. In addition, simply making provenance available to a person does not necessarily constitute an explanation. It would need to be summarised and its essence extracted to be able to construct an explanation addressing a specific regulatory purpose. How we do this is unknown today.

PLEAD brings together an interdisciplinary team of technologists, legal experts, commercial companies and public organisations to investigate how provenance can help explain the logic that underlies automated decision-making to the benefit of data subjects as well as help data controllers to demonstrate compliance with the law. In particular, we will identify various types of meaningful explanations for algorithmic decisions in relation to their purposes, categorise them against the legal requirements applicable to UK businesses relating to data protection, discrimination and financial services. Building on those, we will conceive explanation-generating algorithms that process, summarise and abstract provenance logged by automated decision-making pipelines. An Explanation Assistant tool will be created for data controllers to provision their applications with provenance-based explanations capabilities. Throughout the project, we will engage with partners, data subjects, data controllers, and regulators via interviews and user studies to ensure the explanations are fit for purpose and meaningful. As a result, explanations that are provenance-driven and legally-grounded will allow data subjects to place their trust in automated decisions, and will allow data controllers to ensure compliance with legal requirements placed on their organisations.

Socially-sensitive decisions are made on a daily basis for a wide range of purposes, including credit scoring, insurance and hiring. Increasingly, decision-makers are relying on more advanced automated decisions based on an ever-expanding assortment of personal data. This is because the move towards automated decision-making has various benefits, e.g. the greater speed in which decisions are made or the enrichment of the decision-making process. However, automated decisions are likely to have significant impacts for the individuals who are subject to them (e.g. being refused a mortgage). PLEAD therefore seeks to produce a major contribution in the field of data governance, through the development of techniques and tools exploiting provenance logs, which will allow different types of stakeholders to either produce or receive fit-for-purpose and meaningful explanations each time a decision is automated. As a result, the project will generate a variety of impacts, especially economic and societal impacts.

At a high level, over the long term, the societal impact will be twofold. First, through offering the means to reach a greater degree of transparency and accountability when automated decisions are generated, PLEAD will contribute to improving the quality of life of individuals interacting within smart environments. For instance, when a loan or a mortgage is refused, individuals will have the means to understand the motives of the decision. The same will hold in other sectors such as health, when a decision to put a patient on a waiting list is taken, or justice, when an automated triage system is implemented. Second, PLEAD will also contribute to enhancing the quality of public services, by supporting decision-makers across sectors, including decision-makers operating within public bodies.

The economic impact will benefit both small and medium enterprises and bigger organisations, offering them solutions at the cutting edge of research in order to build ethical and effective governance frameworks for the management of data. This will allow them to have a competitive advantage when investing in innovation and research and when involved in data sharing with partners. Legal compliance will be made easier and thereby cheaper. Customers will be more confident, as they will have the means to remain in control and exercise their right to be heard or invoke human intervention.

At a more granular level, over the short to medium term, the involvement of the industrial partners Roke (law enforcement) and Experian (financial services), and the strategic partnership Southampton CONNECT (smart cities) means that the techniques developed by PLEAD will be tested across different sectors and will feed into the creation of novel services and/or products, benefiting a wide range of stakeholders. Law enforcement agents will be able to test cutting-edge data management tools, intended to enhance auditability and thereby transparency and accountability as well as reliability. Individuals with credit scores will be able to better understand the process of quantification of such scores and either contest or intervene to improve them. Moreover, organisations operating within urban environments concerned about growth and social responsibility such as the city of Southampton will be incentivised to contribute to the setting of an overarching data governance framework promoting transparency and accountability. Citizens will directly benefit from these evolutions through better information and more empowerment.

Ultimately, these experiments will pave the way for the production of best practice, which will be shared with other sectors, and, disseminated to regulators, to inform the release of guidance on the governance of automated decision pipelines.