Making Data Work for Public Sector Policy: A scoping study to develop a mixed-methods framework for culture

Lead Research Organisation: University of Leeds
Department Name: Sch of Performance & Cultural Industries

Abstract

The rationale behind this project is that cultural datasets are currently in a poor state of development and not fit-for-purpose. As many scholars, cultural workers, funders and policymakers have observed, cultural data are incomplete, fragmented and inconsistent. There are no agreed protocols about how to collect data, or even about what kind of data to collect, and no standardised approach to data management. This means that all of the data coming from the cultural sector are not currently set up for cross-analysis, which makes it very difficult to evaluate and provide evidence about the performance and impact of the cultural sector as a whole. This leaves the sector poorly appreciated by the general public, some local councils and even governments, and therefore very vulnerable to public funding cuts in financial and economic crises such as the current COVID-19 crisis.

The central aim of this exploratory project is to co-develop a fit-for-purpose mixed-methods cultural evaluation framework that combines insights from large datasets with soft, qualitative data that capture the lived experiences of people who produce and consume culture, such as artists, curators and audiences. We hope to achieve this by developing a new strategic network to scope and test out possible solutions to the problems listed above though a series of consultations and discussions conducted face-to-face and/or through webinars. This network will include key data, policy and cultural sector stakeholders, including the UK Government's Department for Digital, Culture, Media and Sport (DCMS) and a wide range of national cultural organisations who are committed to developing more robust data standards and protocols and to fostering a more joined-up approach to cultural impact and evaluation by sharing their data and opening them up to research.

A secondary goal of this project is to shape cultural policymaking and in so doing to inform the ways in which public policy in general is developed. We believe that public policymaking is currently too far removed from the people and places it aims to benefit and that is it overly reliant on numbers that fail to tell compelling stories of the cultural sector's impact on society. We hope to demonstrate, therefore, that a more joined-up and people-centred approach to evaluating the impacts of culture on individuals and communities would produce more reliable and credible data that capture impacts in ways that reflect the particular ways in which different people and places experience culture. We will test our emerging data and evaluation frameworks by designing a comprehensive case study of Bradford. We have chosen Bradford essentially because the city is mobilising to apply for UK City of Culture in 2025, and as part of this preparation, it is undertaking a detailed needs analysis that requires complex merging of cultural data with other sources of demographic and health data held by the city. So our engagement with Bradford will be mutually informative.

In order to achieve our objectives, the project has brought together an expert team of academic and cultural sector specialists. Collectively, our researchers have significant experience in research methods, cultural management and evaluation, policy engagement and development, and data management and analysis. Our sector partners include world-leading cultural organisations such as the British Library, Tate and the National History Museum; national cultural funders such as Arts Council England and Historic England; expert cultural data analysts including The Audience Agency and MyCake; and significant regional partners including Bradford 2025, Bradford Teaching Hospitals and its Born in Bradford project, and Bradford City Council. These exceptional partners will work with us on our research journey to bring about a step-change in how cultural data is gathered and used to influence positive social change.

Publications

10 25 50
 
Description We have ascertained that there is consensus about the structural issues with data in the cultural sector as outlined in our original bid (fragmentation, skills gaps in the sector, lack of resources, poor evaluation, etc.). The Covid-19 pandemic highlighted these issues as DCMS was unable to provide an accurate picture of the cultural and creative workforce to Treasury to direct emergency relief funding. We have used this opportunity to work collaboratively with DCMS to try to address in the medium term some of these structural issues.

The project was always designed to be developmental in nature, and we have now successfully scoped out plans for a larger bid to address these issues in an informed way, having consulted and collaborated with key stakeholders over the 15 months of this project. We have realised that more work is needed before we can undertake a case study of Bradford and this has led us to question the utility of a case study approach. We have made significant progress on developing and drafting a Data Framework, which will feed into our future research plans and inform DCMS's future policy directions. We have developed a draft Evaluation Framework which is already attracting interest across the cultural sector and funders/policymakers. We are holding an event at the end of March 2022 to share and refine this framework so will report on the outcome of this next year.
Exploitation Route As described above, the Data Framework will be of great use to cultural funder and policymakers such as Arts Council England and DCMS. It will also help us to establish a national cultural data observatory to improve our understanding of the value and impact of cultural activity and engagement across England. The finding of this project will also have an international significance as other nations look to the UK for best practice on cultural sector data management, analysis and policy.

The Evaluation Framework is likely to be of great use to cultural organisations across the UL and the Centre for Cultural Value will continue to disseminate it via workshops, CPD activities and the development of a dedicated MOOC in 2023.
Sectors Creative Economy,Government, Democracy and Justice,Culture, Heritage, Museums and Collections

 
Description Our findings are feeding into DCMS's plans for improving data collection and analysis in the cultural sector The findings of this research are feeding into the Department in an iterative way via bespoke workshops and meetings with senior DCMS staff and via a dedicated taskforce. .
First Year Of Impact 2001
Sector Creative Economy,Government, Democracy and Justice,Culture, Heritage, Museums and Collections
Impact Types Cultural,Societal,Economic,Policy & public services

 
Description Taskforce membership
Geographic Reach National 
Policy Influence Type Membership of a guideline committee
 
Title Cultural Evaluation Framework 
Description We have developed a new Cultural Evaluation Framework that offers a multidimensional model of how to conduct rigorous evaluation of cultural activities. This will ultimately offer a workable tool for cultural organisations to capture and communicate the value and impacts (social, educational, economic, cultural) of their activities, offering possibilities to develop a stronger evidence base to inform policy development for the cultural sector. 
Type Of Material Improvements to research infrastructure 
Year Produced 2022 
Provided To Others? No  
Impact We are still developing and refining the framework and will publish it later in 2022. 
 
Title Data framework 
Description We have developed a new data framework to guide the future collection, collation and analysis of cultural sector data. This has the potential to transform the ways in which cultural organisations, funders and policymakers gather and use data and to ultimately create a new national dataset that can be opened up for research into the value and impacts of cultural engagement and p[possible correlations with health and wellbeing benefits. 
Type Of Material Improvements to research infrastructure 
Year Produced 2022 
Provided To Others? No  
Impact We are in the final stages of developing and refining this framework and will make it available later in 2022. 
 
Description Arts Council England 
Organisation Arts Council England
Country United Kingdom 
Sector Public 
PI Contribution Research partner/participant
Collaborator Contribution Shared insights into challenges of data collection and analysis on a national scale
Impact Built relationships with Director of Research and discussed plans for a cultural data observatory
Start Year 2018
 
Description Bradford 2025 
Organisation Bradford Institute for Health Research (BIHR)
Department Born in Bradford
Country United Kingdom 
Sector Public 
PI Contribution We advised on embedding cultural value and impact questions into BiB's research programme
Collaborator Contribution Their financial support funded 7 days of artists' time to support interviews and workshops to develop a deeper understanding of the creative methods that artists draw on to elicit impact in arts participants
Impact Their financial support funded 7 days of artists' time to support interviews and workshops to develop a deeper understanding of the creative methods that artists draw on to elicit impact in arts participants
Start Year 2021
 
Description Bradford City Council 
Organisation Bradford Metropolitan District Council
Country United Kingdom 
Sector Public 
PI Contribution We have collaborated with them to scope out a case study of Bradford to inform a future funding bid
Collaborator Contribution They have shared contacts and challenges related to cultural sector data gathering and analysis
Impact Case study scoping - see final report Evaluation framework
Start Year 2021
 
Description Collaborative Partner- DCMS 
Organisation Department for Digital, Culture, Media & Sport
Country United Kingdom 
Sector Public 
PI Contribution Provision of regular one to one meetings, facilitated workshops, and bespoke briefings on research findings and policy implications to DCMS heads of research and ALB network to support evidence based policymaking.
Collaborator Contribution Development of policy placements that can support an ongoing knowledge exchange process between the project and policy makers. Engagement with Centre for Cultural Value policy reference groups Support for collaborative research grant funding applications
Impact Informing the DCMS Review: Valuing culture and heritage capital: a framework towards informing decision making which was acknowledged in their publication of 21 January 2021. Development of collaborative partnerships on data sharing with 10 arms length bodies and the development of a successful application to ESRC resulting in the research project ESRC Research Methods Development Grants: Making Data Work for Public Sector Policy. Has also led to discussions with DCMS on the development of shared evaluation data frameworks.
Start Year 2020
 
Description Collaborative Partner- The Audience Agency 
Organisation The Audience Agency
Country United Kingdom 
Sector Charity/Non Profit 
PI Contribution Support for collaboration with researchers and policy makers informing shaping of participation monitor. Part funding of TAA research activity.
Collaborator Contribution • Design and delivery of demand-side research, exploring the changing needs and habits of the UK population in terms of cultural engagement through COVID. • Making data and insight available to fellow researchers to contextualise their strands of research into the health of the sector and implications for future operation • Ensuring that all research findings from the project are shared and disseminated through TAA online Evidence Hub and through TAA networks of cultural organisations • Convening a group of sector support organisations and working with them to maximise the value of the insights generated by the research in supporting the bounce back of the sector
Impact Design and delivery of participation monitor, exploring the changing needs and habits of the UK population in terms of cultural engagement through COVID. Publication of findings through TAA Evidence Hub and through TAA networks of cultural organisations (UK and international subscriber list is over 15,000 cultural practitioners)
Start Year 2020
 
Description Collaborative Partner: BFI 
Organisation British Film Institute (BFI)
Country United Kingdom 
Sector Charity/Non Profit 
PI Contribution research support to analyse data assessing impacts of Covid-19 on screen industries sector, briefing on emerging findings to support recovery.
Collaborator Contribution Access to UK Screen Sector Task Force (industry and policy leads in the screen sector) for case study respondents
Impact case study
Start Year 2020
 
Description Data research partner LIDA 
Organisation University of Leeds
Department Leeds Institute of Data Analysis
Country United Kingdom 
Sector Academic/University 
PI Contribution We are discussing how we might transfer as much cultural sector data from England as possible to LIDA to establish a national cultural observatory and to open up cultural sector data for research with other data sets such as Public Health England data and retail data
Collaborator Contribution They have advised on data standards and protocols and on hosting costs for large data sets. We have also discussed collaborative PhDs to develop the next generation of cultural data analysts.
Impact Data framework
Start Year 2021
 
Description Creative workshop 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Third sector organisations
Results and Impact As part of Work Strand 2 (the Bradford case study development) a half-day online research workshop (Sandpit) exploring how qualitative data can be better integrated with numerical data to better inform public policy decisions and provide a fuller picture of the value of your work. The workshop was facilitated by an external facilitator and artist and involved a group of 15 researchers, local policymakers and cultural practitioners (freelance artists and workers from local cultural organisations). The workshop revealed significant interest in incorporating creative methods into evaluation but also highlighted the lack of time skills and resources to achieve this within Bradford's current infrastructure. This fed directly into our plans for further research and into the development of the evaluation framework.
Year(s) Of Engagement Activity 2021
 
Description DCMS data taskforce 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Policymakers/politicians
Results and Impact On the back of this research project I was invited to join a new taskforce led by DCMS to explore how we can improve cultural sector data
Year(s) Of Engagement Activity 2022
 
Description Webinar 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Third sector organisations
Results and Impact Harnessing Data for Cultural Productivity and Social Impact: Where Next for the Cultural Sector?
This interactive webinar took place from 1-3pm on 1 February 2022. Hosted by the Centre for Cultural Value it discussed emerging opportunities and challenges for the subsidised cultural sector relating to data and metrics in the light of sectoral needs moving out of the pandemic. The scope was intentionally wide-ranging to provoke novel contributions and to ask how cultural data and evidence can best be integrated into the cultural policy-making process. A briefing slide-pack provided background information, and emerging findings from the CCV's ongoing ESRC-supported project on cultural data policy, was shared with attenders in advance. The following areas were in scope:
- Review of the subsidised cultural sector's data landscape and current challenges
- Prospects for harnessing insight from administrative and statutory data
- Insights from attendance, expenditure and participation data for organisations and funders
- Analytic capacity in the cultural sector
- Longer-term prospects from and implications of the National Data Strategy, Open Data initiatives, use of persistent identifiers in the cultural sector, data pooling, and personal data markets.

A summary of the webinar was written up and fed into the development of the Data Framework.
Year(s) Of Engagement Activity 2022