PryMe, a Universal Framework to Measure the Strength of Privacy-enhancing Technologies

Privacy is a universal value and an important matter of human rights, security, and freedom of expression. However, in the digital era privacy is increasingly becoming eroded, and existing protections in terms of laws and privacy policies turn out to be insufficient because they do not prevent privacy violations from happening. In contrast, privacy protections on a technical level, so-called privacy-enhancing technologies, can prevent privacy violations and are thus a topic of much current research.

One way to show how effective new privacy-enhancing technologies are, i.e. to what extent they are able to protect privacy, is to use privacy metrics to measure the amount of privacy the technologies provide. Even though many privacy metrics have been proposed, there are many studies showing their shortcomings in terms of consistency, reproducibility, and applicability in different application domains. This is an important issue because use of a weak privacy metric can lead to real-world privacy violations if the privacy metric overestimates the amount of privacy provided by a technology.

The proposed research addresses this issue by evaluating the quality of existing privacy metrics, identifying their strengths and weaknesses, and building on this evidence to propose new, much stronger privacy metrics. Our aim is to create novel privacy metrics that measure the effectiveness of privacy-enhancing technologies consistently, reproducibly, and across application domains. To achieve this aim, we will (i) create the modular framework PryMe for the systematic evaluation of privacy metrics, (ii) apply the PryMe framework to evaluate privacy metrics across application domains, and (iii) propose strong new privacy metrics that work in each application domain.

By proposing a single framework to evaluate privacy metrics in many application domains, we allow research ideas on privacy metrics from different domains to complement each other, which will transform how privacy is measured. To further this transformation, we will release open source code for the PryMe framework to enable other researchers to study different application domains and new privacy metrics. In the long term, this will be relevant to improve privacy-enhancing technologies, and thereby improve privacy for end users.

Privacy measurement is important not only to improve privacy-enhancing technologies, but also to analyse trade-offs between privacy and data utility, or between privacy and security. Better privacy metrics therefore not only improve privacy for end users, but also improve the decision-making in situations when privacy needs to be weighed against utility or security. Better privacy metrics can also help improve the user acceptance of new technologies such as vehicular networks and smart homes by showing that privacy issues have been addressed on a technical level.

This project aims to improve the strength of privacy metrics and thus contributes to the foundations and methods used in privacy research. As such, the research will mainly impact academic beneficiaries, especially in the short term. Academic privacy researchers will benefit from this project because they will be able to apply better privacy metrics to show the effectiveness of new privacy-enhancing technologies and use strong metrics to analyse privacy trade-offs. In addition, privacy researchers will benefit from the open source releases of our PryMe framework because they will be able to study other privacy metrics and application domains.

Because this project analyses privacy metrics in four different domains - vehicular networks, smart metering, social networks, and data publishing - a critical component of this project's impact plan is to ensure that academics working in each of the domains can benefit from the research. These researchers will benefit from the availability of strong privacy metrics that have been shown to work well in their domain. To disseminate our research, our academic publishing strategy includes publication venues in each of the domains, and we will organise tutorials at conferences in each of the domains. In addition, we will publish privacy scorecards that summarise the strongest privacy metrics for each domain.

At present, most privacy-enhancing technologies are developed within academia, and only some of them are implemented by companies or in commercial products. We will use two impact activities (workshops with UK and international researchers) to discuss how privacy measurement, and in particular our privacy scorecard, may contribute to a wider commercial uptake of privacy-enhancing technologies.

Beyond academia, the proposed research has the potential for a much wider impact when improved privacy metrics lead to improved privacy-enhancing technologies, and thus improved privacy for users. In the long term, our research will thus benefit the wider public. We will raise awareness of the benefits of our research through public talks and newspaper articles.

In commercial private sector organisations, the current state of privacy measurement focuses on compliance and return-on-investment metrics. We believe that in the long term, these organisations will benefit from including our strong metrics in their privacy assessments. This use of evidence-based privacy metrics may add a unique selling point for any product with privacy implications because the metrics will demonstrate to customers that privacy issues have been addressed on a technical level. In the long term, our privacy metrics can also become part of a Privacy Risks Management Framework, and thus benefit both public and private sector organisations that apply privacy risk management, for example because they handle personal data. Specifically, we will explore how these benefits may be realized with the organisations in the Cyber Security Centre's advisory board (Airbus Group, BT, Deloitte and Rolls Royce), with a view to applying for joint funding from funders with a more applied remit such as Innovate UK, to support our work towards embedding privacy metrics into privacy risks management frameworks.