Co-designing an evaluation framework for design in the context of policy

Lead Research Organisation: University of Brighton
Department Name: School of Art, Design and Media

Abstract

Part of the Cabinet Office's Policy Lab pilot, this project will anchor the Policy Lab's initiative within a strong evaluative framework supported by academic research and co-designed with its major stakeholders. This framework will enable the Lab to evaluate and analyse the impact its design approach has on developing and implementing policy. It will be underpinned by a thorough review of existing literature to ground the Lab's evaluation in established research findings and methods.

The Policy Lab was set up in April 2014 by the Cabinet Office, part of the UK Civil Service. It aims to experiment with bringing design approaches to policy-making in central government by undertaking projects with government departments funding the lab. Led by experienced designer Dr Andrea Siodmok, this is a one-year pilot project that will design services around people's experiences, using data analytics and new digital tools based on policy goals. The Policy Lab replicates similar activities elsewhere notably in Denmark, which set up a cross-ministerial innovation unit MindLab a decade ago, as well as Nesta's Public Innovation Lab and The Australian Centre for Social Innovation.

Most of the existing literature on these initiatives currently belong in reports, conferences, and blog posts; there is comparatively little academic writing about using design approaches in policy making in central government. This project will combine the existing literature on design in policy with related academic research in design studies, participatory design, policy and management research to provide a thorough review of this emerging field.

Informed by this review, the Fellow will work with the Policy Lab and its stakeholders (including government departments) to co-design, iterate and implement a framework to evaluate the success of the Policy Lab's pilot year. The framework will look at how using research methods in the development and implementation of policy makes for better, more relevant, more smoothly implemented policy.

A public blog will be set up on mappingsocialdesign.org to inform audience for relevant stakeholders towards the end of the project to reflect on its findings. Finally, a report aimed at practitioners and managers in policy contexts will be published about the framework and its implications for bringing design approaches into policy-making and delivery.

Planned Impact

This project will lead to academic and societal impact in the short, medium and long term. Four main beneficiary groups will benefit from this research.

1. The primary users and beneficiaries are in the Policy Lab itself, the Open Policy Making team, the wider Cabinet Office. By co-developing an evaluation framework for the Policy Lab, and helping the Policy Lab team adapt and develop its indicators of success, the project will, in the short term, increase the effectiveness of public services, support increased creativity in devising policy in UK central government, and improve the effectiveness of collaboration between the Cabinet Office and government departments.

2. A secondary set of users and beneficiaries are staff in the 17 government departments directly or indirectly involved in supporting the Lab and learning from its activities including the evaluation framework that is the focus of the fellowship. Activities include joint participation in Policy Lab workshops and opportunities to co-design the Policy Lab evaluation framework. Impacts in the medium to long term will be increased effectiveness of public services, increased creativity in devising policy in UK central government, and improved effectiveness in the collaboration between the Cabinet Office and government departments.

3. A wider group of users and beneficiaries in the UK and internationally are those within other central and local government, including other policy labs, and public sector orientated service design initiatives in the UK and internationally such as The Policy Lab, MindLab, and TACSI. Other users include intermediaries such as Nesta, think tanks such as New Economics Foundation. Impacts in the long term include increased effectiveness of public services, and increased creativity when developing public services. Activities include co-hosting or participating in a workshop at the end of the fellowship.

4. A fourth group of users and beneficiaries are designers and managers working in organisations that work with policy-makers and commissioners of public services, including commercial firms such as Capita and Microsoft, design firms such as Futuregov, and NGOs, social enterprises and charities. Impacts in the longer term include increasing the effectiveness of public services, and increased creativity in working with commissioners of public services.

Publications

10 25 50
 
Description This fellowship resulted in my being embedded three days a week for a year in Policy Lab, a small team in the Cabinet Office working with government departments, citizens, experts and specialist design consultancies. I joined about halfway through their first year and was involved actively in helping them design, deliver and make sense of workshops and longer projects that brought new approaches to the work of policy officials. The research involved collaborative discovery about what design approaches could bring to policy making, as the team explored bringing into being new kinds of policy making expertise.

The findings make sense of what the Policy Lab team were doing and their outcomes in the context of central government policy making practice. I combined literatures from design, organisation studies and science and technology studies to summarise their approach and support their ongoing evaluation.

In addition to sharing the findings summarised below, I developed with policy officials a visual evaluation framework informed by organisation studies literature to help make sense of where Policy Lab (and design expertise) can contribute to organisational exploration and to the delivery of policy.

The main findings were:

(1) Policy Lab uses an approach and expertise based in design to help the policy community explore and develop new capabilities in generating and interpreting early-stage insights, engaging with delivery partners, specialists and stakeholders, and closing the delivery gap between policy intent and outcomes.

(2) As well as practically servicing teams on live policy issues, Policy Lab works with departments to develop new hybrid ways of working and to challenge existing ways of doing things in the ongoing mediation between politics, evidence and delivery.

(3) Policy Lab's projects engage a range of people in collective exploration of a policy issue, through which problems and solutions co-evolve in iterative learning cycles.

(4) This approach helps discover and build confidence in plausible insights and concepts that can then further developed and exploited by more conventional means leading to improved delivery and more likely impact.

(5) Insights and concepts generated through this approach reorder what matters, by bringing into view the complexity and lived experiences of people affected by or involved in a policy problem, rather than starting with the existing system.

(6) The approach enables people inside and outside of government to collaborate effectively by enabling more equal participation, generating a shared language and approach and acknowledging difference constructively.
Exploitation Route The findings provide a way to make sense of design in the context of policy making that goes beyond accounts of design activities/methods/outputs.

With ongoing interest in design thinking in business, social innovation and local government, these findings demonstrate what outcomes design approaches can achieve in complex organisations through practically supporting the exploration of opportunities, rather than the current focus on exploiting opportunities.
Sectors Communities and Social Services/Policy,Digital/Communication/Information Technologies (including Software),Government, Democracy and Justice

URL https://researchingdesignforpolicy.wordpress.com
 
Description The fellowship outputs included several means of engaging directly with practitioners, in this context, civil servants. The main outputs were: a research blog; events held at the Cabinet Office; contributions to the Cabinet Office Open Policy Making blog; and a written report. Since there is a lively community of practice exploring what design brings to government innovation active in social media (in particular twitter and blogs), being an active participant in these discussions was another route to impact. Since the original fellowship (2014-15) this field of design practices has continued to grow and alongside it, opportunities for research and engagement with policy makers inside government. Of these channels and outputs, my report about Policy Lab was an initial key output. It was published in October 2015 and shared online as a PDF via my research blog which (by 13 March 2017) had 5656 views and 3612 visitors. Of the 750 printed copies, more than half were distributed in the UK civil service. Other copies were sent to the EU Policy Lab development team, distributed to EU policy officials interested in design attending an event, as well as civil servants and social innovators working on policy issues and academics in the US, Canada, UAE, France, Israel and Australia. The continuing presence of the fellowship blog and my ongoing twitter activity (almost 5000 follower) enable me to continue to be in dialogue with practitioners inside government and other researchers exploring design for policy. At the time of writing I have only anecdotal evidence of how it is influencing civil servants' thinking on what design brings to policy innovation. The policy innovation framework developed in the report resulting from the fellowship has shaped how the Policy Lab team I worked with during the fellowship conceives of its own impact and role and some civil servants are using it to inform the narrative about their activities. Subsequent papers/book chapters based on the research during the fellowship which I have shared via talks, social media and by emailing people directly, has deepened ongoing dialogues with civil servants in the UK and internationally including opportunities to talk at events and be part of private meetings. The research has also lead to a year-long piece of consultancy (2017-18) for the EU Policy Lab, a team within the European Joint Research Centre.
Sector Communities and Social Services/Policy,Government, Democracy and Justice
Impact Types Cultural,Societal,Policy & public services

 
Description Arts and humanities into Policy: Policy maker- researcher speed dating event
Geographic Reach National 
Policy Influence Type Influenced training of practitioners or researchers
URL https://openpolicy.blog.gov.uk/2016/03/01/notes-from-a-policy-speed-dating-event/
 
Description Cabinet Office letter recommending future investment in design research in the UK
Geographic Reach National 
Policy Influence Type Implementation circular/rapid advice/letter to e.g. Ministry of Health
 
Description Open Policy Toolkit published on gov.uk website
Geographic Reach National 
Policy Influence Type Influenced training of practitioners or researchers
URL https://www.gov.uk/guidance/open-policy-making-toolkit
 
Description Short course for Civil Service Learning - Design Thinking for Policy Makers
Geographic Reach National 
Policy Influence Type Influenced training of practitioners or researchers
 
Description Training for civil servants
Geographic Reach Asia 
Policy Influence Type Influenced training of practitioners or researchers
 
Description Arts and humanities into policy making 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Policymakers/politicians
Results and Impact I initiated and organised a workshop held at the Cabinet Office. This brought together civil servants involved in public policy making with arts and humanities researchers to explore possible uses of A&H research to inform policy making, in particular around concepts such as 'community and 'belonging'. The event was then written up in a blog format which was published on the Cabinet Office Open Policy blog. In total 7 policy officials engaged with 11 researchers.
Year(s) Of Engagement Activity 2015
URL https://openpolicy.blog.gov.uk/?s=arts+and+humanities
 
Description Civil Service Policy Profession 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Policymakers/politicians
Results and Impact I took part in a panel discussion on organisational change at the Department of Energy and Climate Change. This was organised by DECC and aimed at UK civil servants involved in developing evidence-based policy making.
Year(s) Of Engagement Activity 2016
 
Description Design Culture Salon, Victoria and Albert Museum, London 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Professional Practitioners
Results and Impact I was one of four speakers at Design Culture Salon on the topic of design innovation. I used Policy Lab's work as an example which generated discussion about how design contributes to the generation and exploration of policy options.
Year(s) Of Engagement Activity 2014
URL http://designculturesalon.org
 
Description Design seminar at Cabinet Office during London Design Festival 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Policymakers/politicians
Results and Impact Policy Lab, a team within the Cabinet Office, organised a small discussion event within London Design Festival 2017, held at 1 Horseguards Road attended by about 25 people including civil servants, researchers, post-graduate students and practitioners. I was invited to share some reflections from being AHRC research fellow in the first year of Policy Lab. This sparked questions and discussion in the room and on twitter.
Year(s) Of Engagement Activity 2017
 
Description Keynote: Council for Higher Education in Art Design conference 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Other audiences
Results and Impact I did the keynote at the annual CHEAD conference which brings together senior managers in art and design institutions. My findings about Policy Lab led to interest about the extent to which design institutions/designers can engage with policy making.
Year(s) Of Engagement Activity 2015
 
Description LabWorks London 2015 - Presentation of fellowship findings 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Policymakers/politicians
Results and Impact I presented findings from my fellowship in Policy Lab at an invite only 'practitioner' day held as part of LabWorks, an event on government innovation held in London and co-organised by Nesta.
Year(s) Of Engagement Activity 2015
URL http://www.nesta.org.uk/event/labworks-2015
 
Description LabWorks conference, London, 2015 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Policymakers/politicians
Results and Impact I chaired a panel discussion of 4 experts discussing the evaluation of innovation initiatives at the LabWorks event held in London co-organised by Nesta.
Year(s) Of Engagement Activity 2015
URL http://www.nesta.org.uk/event/labworks-2015
 
Description Meeting with German civil servant to discuss design for policy 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Policymakers/politicians
Results and Impact Danish Design Centre, the national body promoting design in Denmark, organised a discussion with a senior German civil servant and a manager at Deutsche Telekom to reflect on the use of design approaches within policy development and organisational change. I was one of three people invited to take part, sharing insights from research. The civil servant reported she would like to stay in touch and share reports on her team's work for feedback.
Year(s) Of Engagement Activity 2017
 
Description Open Policy Making blog 
Form Of Engagement Activity Engagement focused website, blog or social media channel
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Policymakers/politicians
Results and Impact I wrote 4 blog posts during my research fellowship on the Cabinet Office Open Policy blog about Policy Lab's activities.
Year(s) Of Engagement Activity 2014,2015,2016
URL https://openpolicy.blog.gov.uk/2015/10/23/panel-discussion-should-policymakers-be-policy-designers-2...
 
Description Open Policy Toolkit 
Form Of Engagement Activity Engagement focused website, blog or social media channel
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Policymakers/politicians
Results and Impact I wrote sections of the first version of the Cabinet Office's Open Policy Toolkit, which was then revised in early 2016 but keeping chunks of the same material.
Year(s) Of Engagement Activity 2015,2016
URL https://www.gov.uk/guidance/open-policy-making-toolkit
 
Description Open event - Cabinet Office - Design for Policy Innovation 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Policymakers/politicians
Results and Impact About 90 people attended an event at the Cabinet Office during which speakers shared perspectives on bringing design approaches to their work in government and complex policy areas. By hosting such an event in the Churchill Room in Whitehall highlighted and brought together a growing network of people involved in design in policy contexts.
Copies of Lucy Kimbell's fellowship report about Policy Lab were distributed at the event.
Year(s) Of Engagement Activity 2015
URL https://openpolicy.blog.gov.uk/2015/10/23/panel-discussion-should-policymakers-be-policy-designers-2...
 
Description Opportunities for policy engagement - Arts and Humanities early career researchers 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Other audiences
Results and Impact I gave a talk about Open Policy Making to a group of early career researchers in the arts and humanities in a programme organised by the Institute for Government.
Year(s) Of Engagement Activity 2015
 
Description Panel discussion, Innovation Growth Lab conference 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Policymakers/politicians
Results and Impact I did a short presentation and took part in a panel discussion about alternatives to randomised control trials in policy making, within a large international conference organised by the Innovation Growth Lab, an organisation co-founded by Nesta, which promotes research into innovation policy.
Year(s) Of Engagement Activity 2016
 
Description Presentation at Danish Design Centre 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Industry/Business
Results and Impact Danish Design Centre organised a one day conference on Experimentation by Design in Copenhagen. My presentation sharing different academic perspectives on experimentation in design sparked questions and discussion, and the slides were shared on the event website and tweeted. The Danish Design Centre reported better understanding among participants of design research. It led to follow up meetings with a Danish foundation seeking input on their work.
Year(s) Of Engagement Activity 2017
URL https://danskdesigncenter.dk/en/events/experimentation-design
 
Description Public event - Cabinet Office - Ethnography in Policy Making 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Policymakers/politicians
Results and Impact 80 people attended a two hour event during which three speakers (and one chair) shared perspectives on the use of ethnographic research in government policy making. This led to new connections/contacts for Policy Lab (the Cabinet Office team which hosted the event) and audiences via social media reporting increased visibility of this expertise.
Year(s) Of Engagement Activity 2015
URL https://openpolicy.blog.gov.uk/2015/03/27/ethnography-in-policymaking/
 
Description SPRU 50th anniversary conference 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Third sector organisations
Results and Impact I took part in a panel discussion about the emergence of 'policy labs' inside government entities, which was an event at the Science Policy Research Unit (SPRU) 50th anniversary conference at Sussex University, which attracted both practitioners and academics including people from government and funders.
Year(s) Of Engagement Activity 2016