How States Account for Failure in Europe (HowSAFE): Risk and the Limits of Governance
Lead Research Organisation:
King's College London
Department Name: Geography
Abstract
What do environmental protection, health care, workplace safety, food, criminal justice, and education share in common? These diverse policy domains are increasingly governed through 'risk-based' approaches to regulation and management. Rather than seeking to eliminate all potential adverse outcomes, risk-based governance involves defining acceptable levels of risk, based on formal assessments of their probability and
consequences, and then focusing control efforts on those risks deemed unacceptable following clearly defined principles. First developed in the fields of environmental health and safety, this risk-based approach has become pervasive, particularly in Anglo-Saxon countries, and is now being promoted internationally as a universal principle for policymaking and implementation, promising a more efficient and rational means of organising governance activities and accounting for their limits and potential failures. However, risk-based approaches embody particular understandings about how the State should define, and account for, adverse governance outcomes and, indeed, the very meaning of governance 'failure' and 'success'. Such ex ante risk-based rationalisations of the limits of governance may conflict with embedded governance traditions, norms, and accountability structures, as well as with deeply held societal values and expectations about how adverse outcomes should be managed, which vary both across countries and policy domains.
To understand the institutional factors shaping the spread and adoption of risk-based governance, HowSAFE uses a comparative case study design focusing on six policy domains-occupational health & safety, flooding, food safety, health care, criminal justice, and education-and four countries-France, Germany, the Netherlands, and UK-to pursue three closely related objectives:
1) To document the extent, and diffusion processes, of risk-based governance across policy domains and national settings in Europe;
2) To compare the design, adaptation and practical application of risk-based instruments to different governance activities and functions, within and between policy domains;
3) To use that comparative dataset to explain the institutional factors driving, shaping, and constraining risk-based governance in Europe, and in so doing reflect more broadly on how states account for failure and the limits of governance.
consequences, and then focusing control efforts on those risks deemed unacceptable following clearly defined principles. First developed in the fields of environmental health and safety, this risk-based approach has become pervasive, particularly in Anglo-Saxon countries, and is now being promoted internationally as a universal principle for policymaking and implementation, promising a more efficient and rational means of organising governance activities and accounting for their limits and potential failures. However, risk-based approaches embody particular understandings about how the State should define, and account for, adverse governance outcomes and, indeed, the very meaning of governance 'failure' and 'success'. Such ex ante risk-based rationalisations of the limits of governance may conflict with embedded governance traditions, norms, and accountability structures, as well as with deeply held societal values and expectations about how adverse outcomes should be managed, which vary both across countries and policy domains.
To understand the institutional factors shaping the spread and adoption of risk-based governance, HowSAFE uses a comparative case study design focusing on six policy domains-occupational health & safety, flooding, food safety, health care, criminal justice, and education-and four countries-France, Germany, the Netherlands, and UK-to pursue three closely related objectives:
1) To document the extent, and diffusion processes, of risk-based governance across policy domains and national settings in Europe;
2) To compare the design, adaptation and practical application of risk-based instruments to different governance activities and functions, within and between policy domains;
3) To use that comparative dataset to explain the institutional factors driving, shaping, and constraining risk-based governance in Europe, and in so doing reflect more broadly on how states account for failure and the limits of governance.
Planned Impact
HowSAFE will contribute to policy practice in UK, France, Germany and the Netherlands, as well as international bodies such as the OECD and EC. The detailed and wider lessons-learned from the case-study research will help foster reflexive learning by policy practitioners across policy domains and country contexts. Given the emphasis on establishing risk-based approaches to governance across the EU, such research has the potential to identify institutional factors that shape or constrain their adoption. For example, the comparative analysis of risk-based concepts that appear to have common application across sectors (e.g. education, food and criminal justice) will help decision-makers to better understand, reappraise, maybe even discard practices enshrined in their respective sectors.
expect to engage with the scholarly community through project working papers and policy reviews that will be made widely available through a project website and disseminated through international conference presentations and peer reviewed publication, including a monograph that synthesizes the cross-country and domain analysis of the project, an edited book on risk-based regulation, and specialist and international academic journal articles.
We will also generate a series of short policy briefs conveying key findings to policy and stakeholder audiences in the four countries, as well as holding three dissemination workshops at the end of the project in Brussels (Maastricht University campus), Paris and Berlin. Those workshops will bring together policy practitioners and selected stakeholders from different domains to discuss HowSAFE and promote reflexive learning across domains. In common with past research experience, we would expect policy interest to go beyond central policy or evaluation units to those in the 'front line' of regulatory enforcement and implementation.
To facilitate both research and impact activities, we will make use of informal project advisory boards in each country comprising invited senior policy practitioners and other experts with an interest in risk-based governance and in-depth knowledge of case study domains. To go beyond contacts made during the research, we will also draw upon existing networks such as: Borraz's presidency of the French Risk and Society Network; the Kings Centre for Risk Management's established network of senior policy and industry contacts; and the ESRC funded Centre for Analysis of Risk and Regulation's established platform.
All project related materials will be made widely available through an actively maintained project website. In addition, an end of project conference in London will bring academic and policy audiences together and we anticipate considerable interest from policy and stakeholder groups.
A minimum of three project advisory board meetings will be held in each country throughout the life of the project both to facilitate the research process and to advise on how to maximise impacts of project findings. Drawing on past experience of ESRC research, regular informal contacts with domain specialist advisory board members can provide invaluable feedback on progress and suggestions for research directions and dissemination routes. We would also hope to feed into, and capitalise upon, ongoing comparative reviews of risk governance in some of the case study countries and international fora.
Dissemination workshops will be held in Paris, Berlin and Maastricht during the third year of the project and a major conference in London will be organised at the end of the project to bring academic and user groups together.
expect to engage with the scholarly community through project working papers and policy reviews that will be made widely available through a project website and disseminated through international conference presentations and peer reviewed publication, including a monograph that synthesizes the cross-country and domain analysis of the project, an edited book on risk-based regulation, and specialist and international academic journal articles.
We will also generate a series of short policy briefs conveying key findings to policy and stakeholder audiences in the four countries, as well as holding three dissemination workshops at the end of the project in Brussels (Maastricht University campus), Paris and Berlin. Those workshops will bring together policy practitioners and selected stakeholders from different domains to discuss HowSAFE and promote reflexive learning across domains. In common with past research experience, we would expect policy interest to go beyond central policy or evaluation units to those in the 'front line' of regulatory enforcement and implementation.
To facilitate both research and impact activities, we will make use of informal project advisory boards in each country comprising invited senior policy practitioners and other experts with an interest in risk-based governance and in-depth knowledge of case study domains. To go beyond contacts made during the research, we will also draw upon existing networks such as: Borraz's presidency of the French Risk and Society Network; the Kings Centre for Risk Management's established network of senior policy and industry contacts; and the ESRC funded Centre for Analysis of Risk and Regulation's established platform.
All project related materials will be made widely available through an actively maintained project website. In addition, an end of project conference in London will bring academic and policy audiences together and we anticipate considerable interest from policy and stakeholder groups.
A minimum of three project advisory board meetings will be held in each country throughout the life of the project both to facilitate the research process and to advise on how to maximise impacts of project findings. Drawing on past experience of ESRC research, regular informal contacts with domain specialist advisory board members can provide invaluable feedback on progress and suggestions for research directions and dissemination routes. We would also hope to feed into, and capitalise upon, ongoing comparative reviews of risk governance in some of the case study countries and international fora.
Dissemination workshops will be held in Paris, Berlin and Maastricht during the third year of the project and a major conference in London will be organised at the end of the project to bring academic and user groups together.
Organisations
Publications
Rothstein H
(2017)
Society, Regulation and Governance: New Modes of Shaping Social Change?
Rothstein H
(2013)
European Parliament Working Group on Risk: a commentary.
in Risk analysis : an official publication of the Society for Risk Analysis
Rothstein H
(2013)
Exploring national cultures of risk governance
in Risk and Regulation
Rothstein H
(2018)
The Routledge Companion to Risk, Crisis and Emergency Management
Rothstein H
(2020)
The boundary conditions for regulation: Welfare systems, state traditions, and the varied governance of work safety in Europe
in Governance
Rothstein, H.
(2021)
COVID19 across 5 continents - Social Challenges, Responses and Consequences
Self D
(2021)
Institutional constraints on 'nudge-style' risk rating systems: explaining why food hygiene barometers were rolled-out in the UK but abandoned in Germany
in Journal of Risk Research
| Description | FINDING 1: Our comparative European research shows that the meanings of, and relationships between, risk and regulation significantly vary according to national traditions and norms of governance in at least three ways (Rothstein et al 2015;2016;2017): i) Constitutional Settings: The utilitarian conceit of risk-based regulation is favoured in polities that emphasise consequentialist justifications of regulatory action but struggles in more rights-based polities that, for example, emphasise equality; ii) Legal Cultures: While anglo-saxon commentators like to portray goals of eliminating risk as hallmarks of continental European precaution, they miss the point that while anglo-saxon countries with common law traditions of legal literalism struggle with goals of zero risk, civil law countries treat headline law as only aspirational and secondary to detailed rules that give effect to such goals; iii) State Traditions: In the UK, risk is invoked to limit regulatory interventions of an otherwise sovereign parliament; in the German Rechtsstaat, risk is invoked as an enabling justification for regulatory interference with constitutionally protected private rights; in the Dutch consociational polity, risk is invoked as a principal for brokering regulatory conflict; while in France risk is invoked to justify interventions that reinforce the legitimacy and protective duties of the State. FINDING 2: Our comparative UK research across policy domains shows that there are at least three institutional preconditions for successful risk-based regulation: i) Regulatory goals need to be unambiguous and stable in order to frame and calibrate regulatory interventions in terms of risk; ii) Trade-offs between regulatory goals have to be amenable to ex ante agreement on defining acceptable risk and there has to be tolerance ex post when adverse outcomes occur; iii) Regulators must be able to reliably assess the probability and consequences of adverse outcomes. Workplace health and safety regulation is one example of meeting such preconditions, having been framed in risk-based terms for over 40 years in the context of stable goals, the common interest of employers and employees in agreeing trade-offs between cost and safety, and a regulator whose expertise is widely commended (Demeritt et al. 2015). Such preconditions have proved harder to meet in flooding, where ex ante agreement on trade-offs can be undermined by political populism when floods occur, and in food safety, where it is hard to reach even ex ante agreement on trade-offs amongst multiple interest groups and consumers. Finally, such preconditions have proved almost impossible in regulating the quality of healthcare and higher education providers in the absence of stable agreement on even the goals of each regime (Beaussier et al 2016). FINDING 3: Big data doesn't necessarily mean better regulation: Our ground-breaking quantitative analyses of enthusiastic attempts by regulators to use statistical surveillance tools to identify risks to healthcare and higher education quality found that big data was at best, of limited value and, at worst, misleading (Griffiths et al 2016). Some of the reasons include the difficulty of finding data that has the right character and is of sufficient reliability to capture the problems addressed by regulators (Beaussier et al 2016). |
| Exploitation Route | There are two principal ways in which our findings should be taken forwards: 1. Our comparative European research on how ideas of risk and regulation are refracted through national traditions and norms of governance have a particular salience in the context of Brexit and the need for government to understand what is at stake in a wide range of risk regulation issues when renegotiating the UK's relationship with the EU. We think that our publications are starting to form the basis for an intellectually novel as well as policy relevant research agenda that could be expanded both in terms of widening the selection of policy domains as well as non-EU country case-studies. 2. Our quantitative statistical analyses that have shown how data-driven approaches to risk-based regulation have failed in healthcare and higher education quality regulation have already led the Quality Assurance Agency and the Care Quality Commission to rethink the way that they prioritise their inspections. We are now building on that novel research and working relationships that we have developed with UK regulators to prepare a new research proposal on the use of big data by regulators as well as in wider government contexts. |
| Sectors | Agriculture, Food and Drink,Chemicals,Construction,Education,Energy,Environment,Financial Services, and Management Consultancy,Healthcare,Government, Democracy and Justice,Manufacturing, including Industrial Biotechology,Pharmaceuticals and Medical Biotechnology,Transport,Other |
| URL | http://www.kcl.ac.uk/sspp/departments/geography/research/Research-Domains/Risk-and-Society/projects/howsafe/index.aspx |
| Description | * IMPACT 1: BIG DATA DOESN'T NECESSARILY MEAN BETTER RISK-BASED REGULATION Our quantitative statistical analyses of enthusiastic attempts by regulators to use statistical surveillance tools to identify risks to healthcare and higher education quality have found that their big data approaches were at best, of limited value and, at worst, misleading (Griffiths et al 2016; Beaussier et al 2016; Griffiths 2017). That research has been ground-breaking by offering the first peer-reviewed assessment of the reliability of big data analytics to improve risk-based regulatory inspection in the UK. As detailed below, our findings have had three substantive impacts on regulatory practice in those two domains as well as in other wider domains as detailed below. Impact 1.1 - Healthcare quality regulation Background: Our study on 'Intelligent Monitoring' - the statistical surveillance system used by the Care Quality Commission (CQC) to identify which NHS Hospital Trusts should be prioritised for inspection and improvement - has had a major impact on the way that the CQC now organises its inspection regime. Intelligent Monitoring was introduced in 2013 in response to the Francis Inquiry's calls on the CQC to improve its approach to inspection following the Mid-Staffs crisis. The system used 150 indicators of care quality as 'smoke detectors' to alert the CQC to poor or declining performance in hospitals. Those indicators were aggregated to generate a numerical 'risk score' to predict which Trusts were most likely to be violating essential care quality standards and help the CQC make better decisions about what, when and where inspections should take place. Our study, which was published in the BMJ Quality & Safety (Griffiths et al 2016), compared the risk scores against the subsequent Ofsted-style quality ratings awarded to NHS Trusts following detailed on-site inspections by large expert teams from the CQC. Our analysis showed that the Intelligent Monitoring risk scores did not predict those inspection-based quality ratings. Nor could the Intelligent Monitoring risk score be used more simply to distinguish the Trusts performing poorly from those Trusts performing well, or even to identify the very worst performing Trusts rated by inspectors as 'Inadequate'. Indeed, according to the measure used by NAO and by the CQC itself, Intelligent Monitoring predictions of hospital quality were wrong more often than they were right. The Impact: In addition to publishing our findings in BMJQ&S, we shared our findings with the CQC and made a submission to the 2016 CQC accountability review conducted by the House of Commons Health Select Committee. The article was retweeted by 50 users placing it in the top 5% of all research outputs scored by Altmetric and was also reported in the weekly journal for healthcare leaders- the Health Service Journal (HSJ 2017). Since then the CQC has replaced Intelligent Monitoring with a new approach called 'Insight', which adopts a different approach to prioritizing its regulatory inspections. Impact 1.2 - Higher Education Regulation Background: Building on the healthcare regulation study, Griffiths completed a related case-studentship PhD jointly funded by the Quality Assurance Agency (QAA) and ESRC under the supervision of Wolf and Rothstein. Following the 2011 White Paper Student at the Heart of the System, the government directed the Quality Assurance Agency to prioritise its inspections of higher education quality based on the continuous monitoring of a basket of indicators. The PhD, therefore, tested whether it was possible to use of the wealth of performance and financial data available in the higher education sector to improve the risk-based prioritisation of QAA regulatory inspections. The study used machine learning to identify the optimal combination of indicators that could reliably predict QAA inspection outcomes across the HE sector. The findings conclusively showed that there was no connection between the available data and the subsequent outcome of QAA inspections, and that it was therefore not possible to use big data to inform a risk-based approach to quality assurance (Griffiths 2017). The Impact: The research is currently being written up for publication, but the results have been widely shared with policy makers including: the QAA; a dedicated high-level seminar at KCL for over 100 senior policy practitioners; the annual European Quality Assurance Forum; and an article in the Times Higher Education (THE 2015). Furthermore, Griffiths provided written and oral evidence to the Business, Innovation and Skills (BIS) House of Commons Select Committee on Assuring Quality in Higher Education inquiry. The QAA has now decided not to use a big data-led approach to prioritising its inspections of HE institutions. Impact 1.3 - Fostering policy learning about big-data and risk-based regulation across the UK and internationally. i. We shared our studies of the use of big data for quality regulation of Healthcare and Higher Education at a dedicated meeting with staff from the Health and Safety Executive and the Health and Safety Laboratory to discuss the reasons for past failures and to help them develop a successful big-data led model for occupational health and safety regulation (June 2016). ii. We convened a dedicated workshop on the use of big data for regulation at the Health and Safety Laboratory with representatives of the Food Standards Agency, Health and Safety Executive, General Medical Council, Defra, Care Quality Commission and the Met Office (January 2017). iii. Rothstein and Griffiths met with the Assistant Deputy Minister at the Ontario Ministry of Government and Consumer Services and the CEO of Ontario's Retirement Homes Regulatory Authority to discuss how they might implement a big data led approach to social care regulation (May 2017). * IMPACT 2: FOSTERING POLICY LEARNING ABOUT RISK-BASED REGULATION THROUGH SECTORAL AND INTERNATIONAL COMPARISON As an integral part of the HowSAFE project we convened a number of workshops for regulators to feedback our findings and foster policy reflection and learning amongst the participants on the use of, and problems posed by, risk-based regulation across Europe. i. March 2014, King's College London HowSAFE organised a one-day workshop bringing together ten UK government agencies and bodies to feedback HowSAFE project findings and compare and learn from their varied approaches to risk-based inspection. Those organisations included: Care Quality Commission; Quality Assurance Agency for Higher Education; Health and Safety Executive; Environment Agency; Defra, Drinking Water Inspectorate; Animal Health and Veterinary Laboratories Agency; Food Standards Agency; Rural Payments Agency; Multi-Agency Public Protection Partnerships (MAPPA); and Huntingdon District Council. ii. May 2015, Sciences-Po (Paris) HowSAFE organised a one-day workshop bringing together food safety regulators from the UK, France, Germany, Netherlands and Luxembourg to compare and better understand their distinctively varied approaches to risk-based food safety inspection. iii. Feb 2015, International Institute of Risk and Safety Management (IIRSM) Rothstein gave a seminar to senior private sector members of IIRSM which drew on HowSAFE project findings (Rothstein et al 2017) on why the UK and continental European member states have adopted different approaches to occupational health and safety regulation and why the concept of risk means different things in those country contexts. iv. June 2015, Professional Standards Authority (PSA) Drawing on HowSAFE project findings, Rothstein gave a talk to the health and social care regulatory bodies on the challenges of risk-based regulation at a specially convened seminar on the topic by the PSA. The PSA's subsequent report 'Right-Touch Regulation-Revised', which sets out the PSA's approach to professional regulation, drew on Rothstein's work and advice. v. Oct 2015, Health and Safety Executive (HSE); June 2016, Health and Safety Laboratory (HSL) Rothstein delivered the HSE Chief Scientist's lecture at the HSE offices in Bootle as well as later at the HSL, which drew on HowSAFE project findings (Rothstein et al 2017) on why the UK and continental European member states have adopted different approaches to occupational health and safety regulation and why the concept of risk means different things in those country contexts. vi. November 2015, Membership of UL's Standards Technical Panel (STP), Canada Invited to be a member of UL's Standards Technical Panel (STP) developing public risk guidelines for Canada. vii. February 2016, Technical Standards and Safety Authority (TSSA) of Canada Conference call with the TSSA to share HowSAFE project findings and to give advice on the problems of implementing risk-based approaches to regulation. viii. May 2016, King's College London HowSAFE organised a one-day workshop bringing together senior representatives from six UK government agencies (including the FSA chief scientist) to compare and learn from their varied approaches to the concept of acceptable risk and HowSAFE project findings. Those organisations included: Health and Safety Executive; Food Standards Agency; Food Standards Agency Northern Ireland; Food Standards Agency Scotland; National Institute for Health and Care Excellence; and the Met Office. REFERENCES Griffiths, A. (2017) Forecasting Failure: Assessing Risks to Quality Assurance in Higher Education Using Machine Learning. PhD. King's College London. Beaussier, A-L., Demeritt, D. Griffiths, A., and Rothstein, H. (2016) 'Accounting for failure: risk-based regulation and the problems of ensuring healthcare quality in the NHS'. Health, Risk and Society 18 (3/4): 205-224. doi: 10.1080/13698575.2016.1192585 Griffiths, A., Beaussier, A-L., Demeritt, D. and Rothstein, H. (2017) 'Intelligent Monitoring? Assessing the Ability of the Care Quality Commission's Statistical Surveillance Tool to Predict Quality and Prioritise NHS Hospital Inspections' British Medical Journal Quality and Safety, 26(2):120-130. doi:10.1136/bmjqs-2015-004687 HSJ (Health Service Journal) (2017) 'CQC risk monitoring 'wrong more than right', study warns' https://www.hsj.co.uk/topics/policy-and-regulation/cqc-risk-monitoring-wrong-more-than-right-study-warns/7016396.article Rothstein, H., Demeritt, D., Paul, R., Beaussier, A-L., Wesseling, M, Howard, M., de Haan, M., Borraz, O., Huber, M., and Bouder, F. (2017) 'Varieties of Risk Regulation in Europe: Coordination, complementarity & occupational safety in capitalist welfare states'. Socio-Economic Review. doi: 10.1093/ser/mwx029 THE (Times Higher Education) (2015) Risk-based quality assessment 'cannot work', study concludes. https://www.timeshighereducation.com/news/risk-based-quality-assessment-cannot-work-study-concludes |
| First Year Of Impact | 2016 |
| Sector | Communities and Social Services/Policy,Education,Healthcare,Other |
| Impact Types | Policy & public services |
| Description | Critique of the Care Quality Commission's 'Intelligent Monitoring' statistical surveillance system |
| Geographic Reach | National |
| Policy Influence Type | Contribution to a national consultation/review |
| Impact | We contributed the findings of our research on the Care Quality Commission's 'Intelligent Monitoring' statistical surveillance to the Health Select Committee's Care Quality Commission Accountability Review. Our research, which was published in BMJ Quality and Safety in 2016, was the first peer-reviewed assessment of the 'Intelligent Monitoring' system which was used by the CQC to monitor health and social care providers and to identify those that should be prioritised for inspection and improvement. Our groundbreaking analysis showed that the Intelligent Monitoring system was unable to detect which hospital trusts were delivering poor quality care, with its predictions of hospital quality being wrong more often than they were right. That research was also reported in a dedicated article in the weekly Health Service Journal (14.3.2017), which is widely read among managers in the NHS. The CQC is now in the process of devising a new and more effective system to monitor health and social care providers. |
| URL | http://qualitysafety.bmj.com/content/26/2/120 |
| Description | Evidence given to HEFCE consultation on Quality Assessment that metrics cannot be used to monitor teaching quality in Higher Education institutions |
| Geographic Reach | National |
| Policy Influence Type | Contribution to a national consultation/review |
| Impact | The HowSAFE project contributed to the HEFCE consultation on 'Future Approaches to quality assessment in England, Wales and Northern Ireland' in June 2015, presenting our research finding that the use of metrics to identify higher education institutions at risk of failing to comply with quality requirements cannot work. Despite much optimism and enthusiastic support within government for data driven approaches for identifying failing higher education institutions, detailed analysis by the HowSAFE project using machine learning tools found that despite exhaustive examination of all possible available data on higher education institutions, there were no metrics that could reliably predict the outcome of QAA reviews. While HEFCE did not publish our (or anyone else's) consultation response, its subsequent publication of its revised operating model for quality assessment indicated that HEFCE now accepted that crude metrics-driven approaches to identifying failing institutions do not work (See paragraph 109, p28 of its 'Revised operating model for quality assessment' published in 2016 [http://www.hefce.ac.uk/media/HEFCE,2014/Content/Pubs/2016/201603/HEFCE2016_03.pdf]): "Throughout this process, the relevant funding body will remain mindful of the complexities involved in making judgements about a higher education provider's performance, and will recognise that data analysis and dialogue in these circumstances needs to be robust, sophisticated and nuanced. It is particularly important to note here that we are not advocating a crude metrics-driven approach, using data to predict providers that might or might not have received successful outcomes under previous quality assessment approaches. Rather, data is used as one source of information to inform a broader judgement supported where needed by suitably qualified and independent experts." |
| URL | http://www.hefce.ac.uk/media/HEFCE,2014/Content/Pubs/2016/201603/HEFCE2016_03.pdf |
| Description | Evidence given to Parliamentary Select committee that metrics cannot be used to monitor teaching quality in Higher Education institutions |
| Geographic Reach | National |
| Policy Influence Type | Contribution to a national consultation/review |
| Impact | Evidence given to the Business, Energy and Industrial Strategy Select Committee that plans by the Quality Assurance Agency to use metrics to identify higher education institutions at risk of failing to comply with quality requirements cannot work. Despite much optimism and enthusiastic support within government for data driven approaches for identifying failing higher education institutions, detailed analysis by the HowSAFE project using machine learning tools found that despite exhaustive examination of all possible available data on higher education institutions, there were no metrics that could reliably predict the outcome of QAA reviews. The QAA is now considering abandoning a metrics driven approach to prioritizing its reviews. |
| URL | https://www.parliament.uk/business/committees/committees-a-z/commons-select/business-innovation-and-... |
| Description | Member of an advisory panel to the Technical Standards and Safety Authority, Toronto, Canada on developing a national standard on public risk management. |
| Geographic Reach | North America |
| Policy Influence Type | Participation in a guidance/advisory committee |
| Description | Member of the UK Department for Business, Energy and Industrial Strategy's Better Regulation Executive's Network of Experts |
| Geographic Reach | National |
| Policy Influence Type | Participation in a guidance/advisory committee |
| Description | Kings Together Multi and Interdisciplinary seed fund |
| Amount | £20,000 (GBP) |
| Funding ID | Big Data for Better Regulation or Better Regulation for Big Data? An Exploratory Analysis of Algorithmic Regulation in British Government |
| Organisation | King's College London |
| Sector | Academic/University |
| Country | United Kingdom |
| Start | 01/2017 |
| End | 07/2017 |
| Description | Wellcome Trust Seed Award |
| Amount | £93,000 (GBP) |
| Funding ID | Can big data improve healthcare quality regulation? An international comparative analysis |
| Organisation | Wellcome Trust |
| Sector | Charity/Non Profit |
| Country | United Kingdom |
| Start | 02/2018 |
| End | 01/2019 |
| Description | 'Paris Risk Group' Workshop, Food Standards Agency, London March 2014 |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Policymakers/politicians |
| Results and Impact | Rothstein's talk 'Exploring national cultures of risk governance' stimulated considerable interest, both amongst academics as well as the numerous policy professionals in attendance. I have had contacts with policy officials from the HSE since my talk to discuss how to pursue some of the themes in greater depth and reflect on the policy implications of the research. |
| Year(s) Of Engagement Activity | 2014 |
| Description | ECPR Standing Group for Regulatory Governance Conference, Barcelona, June 2014 |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Type Of Presentation | paper presentation |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | We organised a panel dedicated to the HowSAFE project comprising five papers given by members of the UK and other partner teams. The panel sparked questions and discussion afterwards. Talks by the UK project team included: Howard M, Rothstein H, Demeritt D, Beaussier A-L. 2014. Mobilizing risk: explaining policy transfer in food and occupational safety regulation in the UK Demeritt D 2014. Risk Governance: From Governing Risk to Governing Through Risk The discussion influenced the ideas of other academics. |
| Year(s) Of Engagement Activity | 2014 |
| Description | ECPR Standing Group for Regulatory Governance Conference, Tilberg, NL, July 2016 |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Presented a paper entitled 'The welfare state as a boundary condition for the regulatory state: The case of occupational health and safety' to a session on the regulator welfare state. The paper prompted questions and considerable discussion. |
| Year(s) Of Engagement Activity | 2016 |
| Description | ECPR Standing Group for Regulatory Governance Conference, Tilberg, NL, July 2016 |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Policymakers/politicians |
| Results and Impact | In addition to giving individual papers throughout the conference, we organised a panel dedicated the HowSAFE project comprising four papers given by members of the UK and other partner teams. The panel sparked questions and discussion afterwards. |
| Year(s) Of Engagement Activity | 2016 |
| Description | Exploring Varieties of Risk Regulation in Europe, Health and Safety Executive, Bootle, October 2015 |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Policymakers/politicians |
| Results and Impact | Rothstein was invited to give a talk to HSE staff on the different regulatory rationales underpinning the implementation of risk concepts and tools in occupational health and safety regulation across Europe. There was much discussion of the extent to which the explanations offered in the talk can help explain significant differences in European attitudes and understandings of risk. |
| Year(s) Of Engagement Activity | 2015 |
| Description | Flooding workshop, Defra, London, December 2013 |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Policymakers/politicians |
| Results and Impact | Rothstein contributed to an extensive discussion of flood insurance and individual's actions. Continued contact with Defra officials on flood risk management. |
| Year(s) Of Engagement Activity | 2013 |
| Description | Global Governance Workshop, Maxwell School of Syracuse University, New York, June 2013 |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Policymakers/politicians |
| Results and Impact | Rothstein's talk on the HowSAFE project sparked questions and discussion afterwards Talk sparked questions and discussion afterwards |
| Year(s) Of Engagement Activity | 2013 |
| Description | HowSAFE / FSA / HSE Workshop on the Acceptability of Risk, King's College London, May 2016 |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Policymakers/politicians |
| Results and Impact | The HowSAFE project organised with the Health and Safety Executive and Food Standards Agency a one day workshop on the Acceptability of Risk, which was hosted at KCL. Attendance included senior regulators and scientists from the UK Food Standards Agency, the Health and Safety Executive, National Institute for Clinical Excellence and Met Office, |
| Year(s) Of Engagement Activity | 2016 |
| Description | HowSAFE / HSE / HSL / FSA Workshop on Big Data and Predictive Analytics Workshop for Risk Regulators, Health and Safety Laboratory, Buxton, Jan 2017 |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Policymakers/politicians |
| Results and Impact | The HowSAFE project organised a one day workshop jointly with the UK Health and Safety Laboratory, Health and Safety Executive and Food Standards Agency on the use of Predictive Analytics by risk regulators. The workshop was hosted by HSL, Buxton, and the contributors included senior regulators and scientists from the UK Health and Safety Laboratory, the Health and Safety Executive, Care Quality Commission, Defra, General Medical Council, Met Office. |
| Year(s) Of Engagement Activity | 2010,2017 |
| Description | HowSAFE Food Safety Inspection Workshop, Sciences Po, Paris, May 2015 |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | The HowSAFE team organised a workshop on risk-based food safety inspection and enforcement in Paris. Food safety regulators from the UK, France, Netherlands, Germany and Luxembourg gave presentations on the different ways in which they had implemented risk based food safety inspection and enforcement. Strikingly different practices were observed, which stimulated considerable discussion and is being written up as a research article. |
| Year(s) Of Engagement Activity | 2015 |
| Description | HowSAFE Workshop on Risk and Regulation, Sciences Po, Paris, September 2017 |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | HowSAFE organised a small workshop for academics and professional practitioners on risk and regulation in the EU. |
| Year(s) Of Engagement Activity | 2014,2017 |
| Description | HowSAFE workshop on Law, Regulation and Governance at Bielefeld University, Germany. November 2014. |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Type Of Presentation | paper presentation |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Rothstein's presentation to a HowSAFE organised international workshop on risk and regulation on 'When 'Never' Means 'Sometimes': Explaining the Varied Uptake of 'Risk' Concepts in Regulation across the EU' sparked questions and discussion afterwards. Influenced the ideas of other academics |
| Year(s) Of Engagement Activity | 2014 |
| Description | HowSAFE workshop on Risk-based Inspection, King's College London, March 2014 |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Policymakers/politicians |
| Results and Impact | The HowSAFE team organised a workshop on risk-based regulatory inspection at KCL. Contributors included regulators and representatives from the UK Health and Safety Executive, Environment Agency, Food Standards Agency, Drinking Water Inspectorate, Care Quality Commission , Rural Payments Agency, Animal Health and Veterinary Laboratories Agency, Human Tissue Authority, Quality Assurance Agency for Higher Education, Multi-Agency Public Protection Partnerships (MAPPA) and Huntingdon District Council. Strikingly different practices were observed, which stimulated considerable discussion and informed subsequent research on risk-based inspection by the regulator for health and social care - the Care Quality Commission - and the regulator for Higher Education quality - the Quality Assurance Agency. That later phase of research resulted in significant and hitherto unknown insights into the fundamental problems facing risk-based inspection for health care (published in the journals: BMJ Q&S, and Health, Risk and Society) and higher education. |
| Year(s) Of Engagement Activity | 2014 |
| Description | International Workshop on Risk and Psychiatry, King's College London, June 2013 |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | The HowSAFE team organised and hosted a small international workshop on risk and psychiatry at King's in June 2013, in collaboration with other colleagues - Prof George Szmuckler and Prof Nik Rose - from across KCL. Professional psychiatrists were invited from France, Germany and the Netherlands to talk and reflect upon the varied ways in which risk concepts and risk management practices have permeated psychiatry across those countries in comparison with the UK. Some striking patterns were observed and discussions are ongoing as to how to progress these ideas in further research. |
| Year(s) Of Engagement Activity | 2013 |
| Description | Invited presentation to the Computers, Privacy and Data Protection (CPDP) Conference, Brussels, January 2016 |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Invited presentation on 'The Challenges of Risk-Based Regulation' |
| Year(s) Of Engagement Activity | 2016 |
| Description | Invited presentation to the UK Health and Safety Laboratory, Buxton, June 2016 |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Policymakers/politicians |
| Results and Impact | Rothstein was invited to give a talk to HSL staff on the different regulatory rationales underpinning the implementation of risk concepts and tools in occupational health and safety regulation across Europe. There was much discussion of the extent to which the explanations offered in the talk can help explain significant differences in European attitudes and understandings of risk. |
| Year(s) Of Engagement Activity | 2016 |
| Description | Invited talk at the International Institute of Risk and Safety Management, London, Feb 2015 |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | I gave a talk on the way in which risk ideas are differently conceived within occupational health and safety regulation across EU countries. The industry practitioners attending the talk were very interested insofar as it helped explain their different experiences of other EU member states and helped them think about how their firms should think about the issues in future. |
| Year(s) Of Engagement Activity | 2015 |
| Description | Invited talk to the Professional Standards Association Workshop, London, June 2015 |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | Rothstein gave a talk to a workshop hosted by the Professional Standards Association on the challenges of risk-based regulation of professionals. The talk stimulated a lot of discussion and interest. |
| Year(s) Of Engagement Activity | 2015 |
| Description | Invited talk to the Professional Standards Association Workshop, London, March 2015 |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | Rothstein gave a talk on the way in which risk ideas are differently conceived within regulation across EU countries. The audience attending the talk were very interested and it resulted in a invitation to give a further talk to the PSA in June 2016 and the Health and Safety Executive in October 2015. |
| Year(s) Of Engagement Activity | 2015 |
| Description | News article in Health Service Journal about our research: 'CQC risk monitoring 'wrong more than right', study warns', 14 March 2017 |
| Form Of Engagement Activity | A press release, press conference or response to a media enquiry/interview |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | News article reporting the findings of our paper published in BMJ Q&S (Griffiths et al 2017) which showed that the statistical surveillance tool used by the Care Quality Commission to prioritise its inspections of acute hospital trusts does not work. |
| Year(s) Of Engagement Activity | 2017 |
| URL | https://www.hsj.co.uk/topics/policy-and-regulation/cqc-risk-monitoring-wrong-more-than-right-study-w... |
| Description | Regulation in Crisis Workshop, CARR, LSE, June 2016 |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Other audiences |
| Results and Impact | Presented a paper entitled ' Did we do enough to understand what was at stake in EU conflicts over risk and regulation?' to an ESRC sponsored workshop on Regulation in Crisis Workshop, CARR, LSE. |
| Year(s) Of Engagement Activity | 2016 |
| Description | Regulation in Crisis? ESRC Seminar Series at the Centre for Analysis of Risk and Regulation. LSE. December 2014. |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Rothstein's talk on the HowSAFE project sparked questions and discussion afterwards amongst policymakers and academics. My talk on the HowSAFE project sparked questions and discussion afterwards amongst policymakers and academics. |
| Year(s) Of Engagement Activity | 2014 |
| Description | Risk Futures: Advances in Understanding, Calculating and Managing Risk, Durham University, November 2014 |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Rothstein gave a paper entitled: Risk and the state: explaining different national responses to risk concepts and tools. The talk solicited interest from representatives of the insurance industry and security services. |
| Year(s) Of Engagement Activity | 2014 |
| Description | Roundtable meeting at the Care Quality Commission to discuss HowSAFE project's analysis of CQC's statistical surveillance systems, London, June 2015 |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Policymakers/politicians |
| Results and Impact | Rothstein, Demeritt and Griffiths had a roundtable discussion with the senior staff at the Care Quality Commission (including a representative of the Department of Health) on 19 June 2015 to discuss the findings of their analysis of the effectiveness of the CQC's risk-based approach to prioritising inspection of acute hospital trusts. That analysis had found that the neither the CQC's old approach of Quality Risk Profiles nor the current approach of Intelligent Monitoring could effectively prioritise inspection activity. It was agreed that the results were significant and that the CQC needed to develop a new approach to prioritising inspection activity, which the CQC has subsequently been seeking to do. A revised version of the original analysis is forthcoming in the journal BMJ Quality and Safety in 2016. |
| Year(s) Of Engagement Activity | 2015 |
| Description | Society for Risk Analysis, Arlington, VA, USA |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Policymakers/politicians |
| Results and Impact | We organised a panel dedicated the HowSAFE project comprising four paper given by members of the UK and other partner teams. The panel sparked questions and discussion afterwards. |
| Year(s) Of Engagement Activity | 2015 |
| Description | Society for Risk Analysis-Europe, Maastricht, NL |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Policymakers/politicians |
| Results and Impact | In addition to giving individual papers throughout the conference, we organised a panel dedicated the HowSAFE project comprising four paper given by members of the UK and other partner teams. The panel sparked questions and discussion afterwards. |
| Year(s) Of Engagement Activity | 2015 |
| Description | Why Risk-Based Quality Assurance Reforms Can't Work. King's College London, November 2015. |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Policymakers/politicians |
| Results and Impact | HowSAFE jointly hosted a seminar on 30 November 2016 with the International Centre for University Policy Research at KCL to disseminate the findings of a major comprehensive study by a KCL PhD student Alex Griffiths under the supervision of Henry Rothstein and Alison Wolf, which has has concluded that data-driven, risk-based approaches to assessing quality in Higher Education are neither accurate nor practically feasible. Speaking to a large audience of 50 high-level policy officials, practitioners and academics, the research found that there are no metrics that can reliably predict the outcome of QAA reviews. Alex was asked to present his findings to the BIS select committee and the research has attracted considerable publicity, not least in the Times Higher Education. |
| Year(s) Of Engagement Activity | 2015 |