The Influence of Widening Participation Evaluation in Higher Education

Following Aim higher (2004-2011), widening participation (WP) departments in universities across England have been developing frameworks and tracking students to evaluate the impact of their WP outreach delivery (Bowes et al, 2015; Hayton & Bengry-Howell, 2016). The trend for more robust evaluation (qualitative and quantitative) has continued, and in 2017, the Office for Fair Access (OFFA) proposed standards for the evaluation of outreach interventions (Crawford et al, 2017). Yet, whilst focus on evaluation is growing in the Higher Education (HE) sector, there remains a gap in research exploring the influence of evaluation approach, methods and findings on WP practice and policy (Henry and Mark, 2003, Butcher et al, 2012). To address this gap, in collaboration with OFFA, this research aims to investigate the influences of WP evaluation on outreach practice and policy.
Proposed research questions (RQs):

1. How has HE widening participation evaluation been conceptualised and designed at the institutional and national levels post-Aim higher?
2. From the perspective of HE WP practitioners and national policy makers, what is the influence of evaluation approach, methods and findings on HE WP practice and policy at the individual (practitioner), institutional and national levels?

Stage 1: Document analysis and literature review

This stage will address RQ1 and at the institutional level will include a document analysis of a representative sample of ten universities' access agreements since 2015. To identify a heterogeneous group, English HE institutions will be selected using proportionate stratified sampling based upon institution type (Patton, 1990). Access agreements are working documents that HE institutions use to report to OFFA on the support they provide and evaluation outcomes of access to HE and retention within HE for under-represented groups (Bowes et al, 2013). A thematic analysis of each agreement will be conducted, allowing themes to emerge from the data (Bowen, 2009). Furthermore, to guide and support the thematic analysis of institutional access agreements, literature on WP evaluation (at the national level) published post-Aim higher will be synthesised, including peer-reviewed articles, white papers, and government reports. This additional analysis will provide further understanding of the theoretical underpinnings of WP evaluation. Using three years of access agreements will allow for the tracking of changes in the conceptualisation of WP evaluation and practice over time (ibid). OFFA will provide access agreement data for the document analysis.

Stage 2: Interviews with WP practitioners and policy-makers

This stage addresses RQ2. It will include in-depth interviews with thirty WP practitioners from the stratified sample of ten HE institutions (three per institution), three national policy-makers from the Office for Students and three from the Department for Education. Literature regarding pathways to impact, or what Henry & Mark (2003) refer to as "influence pathways", will inform the interview protocol and questions. Evaluation influence can be conceptual, instrumental and symbolic (Henry & Mark, 2003). Therefore, to allow for the exploration of emergent theoretical constructs regarding evaluation influence at individual and interpersonal levels (Starks & Trinidad, 2007), thematic analysis of the data will be employed (Ritchie & Lewis, 2014). For example, to identify how evaluation has influenced practitioner's behaviour and attitudes towards WP practice. Or how discussing the process or outcomes of WP evaluation with other practitioners in the field has influenced interpersonal rhetoric about WP practice and evaluation (Henry & Mark, 2003). To reduce biases and corroborate the findings, interview data will be triangulated with data from the document analysis (Bowen, 2009).