Re-thinking impact, evaluation and accountability in youth work
Lead Research Organisation:
King's College London
Department Name: Education,Communication & Society
Abstract
The proposed research will investigate the policy and practice of evaluation and accountability in youth work. It will collaborate with young people, youth workers, managers, funders and policy makers/influencers, to understand the effects of impact measurement, and develop approaches to evaluation that are congruent with youth work practice.
Youth work can be life-changing for marginalised young people, developing social and cultural opportunities and contributing to wider community benefits. It takes place in youth clubs, community buildings and on the streets, and is open to all, or aimed at groups with shared experiences of oppression (e.g. young women, LGBT young people, young refugees). It is rooted in the 'voluntary principle' (i.e. that it is young people's choice to take part), enabling a more equal power dynamic than that in schools and formal social services.
In recent years, spending reductions have led to the closure of many local government youth services and charitable projects. In this context, a 2011 House of Commons Education Select Committee inquiry asked youth organisations in England to demonstrate the value of their work by developing clearer outcomes measures. In response, a new youth impact agenda has emerged, instigated by government and influential third sector organisations, to promote the measurement of impact in youth settings.
While the youth impact agenda has been welcomed by many youth sector bodies, some practitioner organisations have argued that measuring predefined outcomes is inconsistent with youth work. Impact is associated with the quantitative measurement of planned outcomes, such as through 'pre and post' tests or randomised control trials. However, personal and social development are notoriously difficult to measure, and impact tools themselves can be inappropriate in youth work settings: for example, questionnaires asking young people about their attitudes or behaviour can be intrusive, particularly early on in a young person's engagement. Impact measurement is particularly challenging in community-based open youth work settings, where outcomes emerge gradually over time rather than being pre-defined.
The youth impact agenda has quickly become influential, yet we know little about how it has been implemented in open youth work settings, or what effects it has had on practice. This is important: if funding is tied to quantitative measures of impact, then it will be difficult for open youth work to demonstrate its value according to those measures, and a valuable tradition of practice built up over more than a century might be lost. Even less is known about how young people experience and perceive impact mechanisms; young people are not a homogeneous group, and it could be that some groups of young people gain satisfaction from reflecting on their outcomes, while others might find impact-related questioning intrusive and off-putting.
This three-year research project will involve 146 participants in qualitative interviews, focus groups, participant observations and youth participatory methods (photography, peer interviewing and film-making). It will find out how the youth impact agenda is implemented in practice, and how impact processes are experienced and perceived by young people and youth workers. It will include the perspectives of funding agencies and policy makers, and explore how and why 'youth impact' has become so important at this time. Research findings will be shared in peer-reviewed journals, conference presentations, and a symposium to discuss policy and impact in the wider social sector. Findings will be disseminated widely through a short film (made and disseminated by young people); a series of deliberative discussions involving practitioners, managers, funders and policy makers/influencers; policy and practice briefings; and a practitioner conference to share approaches to evaluation and accountability that are congruent with open youth work.
Youth work can be life-changing for marginalised young people, developing social and cultural opportunities and contributing to wider community benefits. It takes place in youth clubs, community buildings and on the streets, and is open to all, or aimed at groups with shared experiences of oppression (e.g. young women, LGBT young people, young refugees). It is rooted in the 'voluntary principle' (i.e. that it is young people's choice to take part), enabling a more equal power dynamic than that in schools and formal social services.
In recent years, spending reductions have led to the closure of many local government youth services and charitable projects. In this context, a 2011 House of Commons Education Select Committee inquiry asked youth organisations in England to demonstrate the value of their work by developing clearer outcomes measures. In response, a new youth impact agenda has emerged, instigated by government and influential third sector organisations, to promote the measurement of impact in youth settings.
While the youth impact agenda has been welcomed by many youth sector bodies, some practitioner organisations have argued that measuring predefined outcomes is inconsistent with youth work. Impact is associated with the quantitative measurement of planned outcomes, such as through 'pre and post' tests or randomised control trials. However, personal and social development are notoriously difficult to measure, and impact tools themselves can be inappropriate in youth work settings: for example, questionnaires asking young people about their attitudes or behaviour can be intrusive, particularly early on in a young person's engagement. Impact measurement is particularly challenging in community-based open youth work settings, where outcomes emerge gradually over time rather than being pre-defined.
The youth impact agenda has quickly become influential, yet we know little about how it has been implemented in open youth work settings, or what effects it has had on practice. This is important: if funding is tied to quantitative measures of impact, then it will be difficult for open youth work to demonstrate its value according to those measures, and a valuable tradition of practice built up over more than a century might be lost. Even less is known about how young people experience and perceive impact mechanisms; young people are not a homogeneous group, and it could be that some groups of young people gain satisfaction from reflecting on their outcomes, while others might find impact-related questioning intrusive and off-putting.
This three-year research project will involve 146 participants in qualitative interviews, focus groups, participant observations and youth participatory methods (photography, peer interviewing and film-making). It will find out how the youth impact agenda is implemented in practice, and how impact processes are experienced and perceived by young people and youth workers. It will include the perspectives of funding agencies and policy makers, and explore how and why 'youth impact' has become so important at this time. Research findings will be shared in peer-reviewed journals, conference presentations, and a symposium to discuss policy and impact in the wider social sector. Findings will be disseminated widely through a short film (made and disseminated by young people); a series of deliberative discussions involving practitioners, managers, funders and policy makers/influencers; policy and practice briefings; and a practitioner conference to share approaches to evaluation and accountability that are congruent with open youth work.
Planned Impact
1) Young people, particularly those who participate in youth work. Youth work is usually open to all young people of around 11-19 years in age, and is often attractive to and aimed at groups who are marginalised or disadvantaged by mainstream services.
1a) During the project: 60 young people will take part in interviews and/or focus groups, enabling them to reflect on their journeys through youth work and their experiences of youth work monitoring, measurement and evaluation. 20 young people will be involved in participatory youth research, including peer interviewing and photo-voice methods. 10 will plan, make and evaluate a film, drawing on the research findings, which they will use as the basis of workshops in other youth clubs. An additional 45 (3 groups of 15) will take part in these workshops, discussing and debating youth work's value and how it is / should be evaluated. Over 100 young people will view the film on social media. 2 young adults (recent youth work participants) will be on the Advisory Group.
1b) After the project: it is hoped that thousands of young people taking part in youth services in the UK and internationally will benefit through the project's contribution to better evidenced services, more congruent evaluation and accountability mechanisms, and more youth-centred evaluation in youth work.
2) Youth workers, managers, youth organisations (local authority, voluntary sector, social enterprises).
2a) During the project: 12 youth organisations (8 UK, 4 USA) will participate directly in the research. 42 youth workers, 12 managers and 12 administrators will take part in interviews / focus groups, providing an opportunity to reflect on the value and impact of youth work and its evaluation. Two in-depth case study organisations and their employees will benefit additionally from more in-depth discussions on evaluation, and from collaborating with the research team on youth participatory research. 70 practitioners and managers will benefit through involvement in round-table workshops and the practitioner conference. 3 experienced youth workers will be on the Advisory Group. Over 200 practitioners and managers from the voluntary sector, public sector and social enterprises will view the young people's film, and read accessible practice briefings, blog posts, short articles, and open access academic publications.
2b) After the project: it is hoped that hundreds of youth workers, managers and organisations will have more awareness of, and confidence to use, youth-centred and practice-oriented approaches to evaluation, and that they will benefit from more awareness amongst funders and policy makers of youth-centred evaluation methods.
3) Policy makers / influencers, funding agencies, third sector organisations in the areas of youth work, youth impact or charitable / public service impact. These groups overlap in the youth sector, where key charities and social enterprises are particularly influential on policy and practice.
3a) During the project: 4 representatives of influential third sector organisations, 2 with significant policy influence, will be on the advisory group. 12 policy makers / advisors (UK) and 8 experts / informants (USA) will be interviewed, providing opportunities for reflection on the policy process around youth impact, and how youth impact is perceived and experienced in youth work settings. 12 funder representatives and 12 policy makers / influencers will take part in round-table discussions to develop alternative approaches to impact. 3 from each group will be brought together with practitioners and managers for a final round-table to discuss ways forward. The research team will publish and disseminate short policy briefings, and will be prepared to contribute to existing and unanticipated policy processes.
3b) After the project: It is hoped that these groups will be more informed on using practice-oriented, youth-centred approaches to accountability and evaluation.
1a) During the project: 60 young people will take part in interviews and/or focus groups, enabling them to reflect on their journeys through youth work and their experiences of youth work monitoring, measurement and evaluation. 20 young people will be involved in participatory youth research, including peer interviewing and photo-voice methods. 10 will plan, make and evaluate a film, drawing on the research findings, which they will use as the basis of workshops in other youth clubs. An additional 45 (3 groups of 15) will take part in these workshops, discussing and debating youth work's value and how it is / should be evaluated. Over 100 young people will view the film on social media. 2 young adults (recent youth work participants) will be on the Advisory Group.
1b) After the project: it is hoped that thousands of young people taking part in youth services in the UK and internationally will benefit through the project's contribution to better evidenced services, more congruent evaluation and accountability mechanisms, and more youth-centred evaluation in youth work.
2) Youth workers, managers, youth organisations (local authority, voluntary sector, social enterprises).
2a) During the project: 12 youth organisations (8 UK, 4 USA) will participate directly in the research. 42 youth workers, 12 managers and 12 administrators will take part in interviews / focus groups, providing an opportunity to reflect on the value and impact of youth work and its evaluation. Two in-depth case study organisations and their employees will benefit additionally from more in-depth discussions on evaluation, and from collaborating with the research team on youth participatory research. 70 practitioners and managers will benefit through involvement in round-table workshops and the practitioner conference. 3 experienced youth workers will be on the Advisory Group. Over 200 practitioners and managers from the voluntary sector, public sector and social enterprises will view the young people's film, and read accessible practice briefings, blog posts, short articles, and open access academic publications.
2b) After the project: it is hoped that hundreds of youth workers, managers and organisations will have more awareness of, and confidence to use, youth-centred and practice-oriented approaches to evaluation, and that they will benefit from more awareness amongst funders and policy makers of youth-centred evaluation methods.
3) Policy makers / influencers, funding agencies, third sector organisations in the areas of youth work, youth impact or charitable / public service impact. These groups overlap in the youth sector, where key charities and social enterprises are particularly influential on policy and practice.
3a) During the project: 4 representatives of influential third sector organisations, 2 with significant policy influence, will be on the advisory group. 12 policy makers / advisors (UK) and 8 experts / informants (USA) will be interviewed, providing opportunities for reflection on the policy process around youth impact, and how youth impact is perceived and experienced in youth work settings. 12 funder representatives and 12 policy makers / influencers will take part in round-table discussions to develop alternative approaches to impact. 3 from each group will be brought together with practitioners and managers for a final round-table to discuss ways forward. The research team will publish and disseminate short policy briefings, and will be prepared to contribute to existing and unanticipated policy processes.
3b) After the project: It is hoped that these groups will be more informed on using practice-oriented, youth-centred approaches to accountability and evaluation.
Organisations
Publications
De St Croix T
(2023)
'It's a great place to find where you belong': creating, curating and valuing place and space in open youth work
in Children's Geographies
De St Croix T
(2020)
Re-imagining accountability: storytelling workshops for evaluation in and beyond youth work
in Pedagogy, Culture & Society
De St Croix T
(2022)
'Capturing the magic': grassroots perspectives on evaluating open youth work
in Journal of Youth Studies
Doherty L
(2019)
The everyday and the remarkable: Valuing and evaluating youth work
in Youth and Policy
Louise D
(2021)
There's a cupboard full of pasta! Beyond sustenance: reflections on youth work and commensality
in Concept: The Journal of Contemporary Community Education Practice Theory
| Title | Value of Youth Work |
| Description | Film co-created by young people inspired by and in conversation with the research. In the film, young people from three youth clubs share their experiences and perspectives on the value of youth work. Young people planned, filmed, took part, and contributed to editing, supported by community and youth film company, 'Mouth That Roars'. |
| Type Of Art | Film/Video/Animation |
| Year Produced | 2021 |
| Impact | Youth workers have shared the film widely including in workshops with young people. The film has been shared at several events for practitioners and policy. Youth workers and young people have found that the film clearly advocates for the value of youth work, and is useful in raising awareness of youth work and its value to young people and their lives. |
| URL | https://vimeo.com/597256970 |
| Description | 1) There are significant tensions in policy and practice between the requirements and norms of conventional outcomes-based impact measurement, and the needs of youth work (particularly grassroots and open youth work, especially when this is undertaken by smaller and community-based organisations). At a policy and strategic level, an initial imperative for youth organisations to engage in conventional impact measurement has generally developed into a more nuanced approach. However, there remains an implied or stated hierarchy of evidence that often sits uneasily with youth work practice. Evaluation that informs and develops practice (e.g. session debriefs, reflective meetings and young people's verbal and creative feedback) is not always perceived in youth organisations as being valued or 'counted' by funders and in policy. 2) Impact measurement, monitoring and evaluation infuse the daily life of youth work settings in a variety of forms, appearing taken-for-granted and necessary, whilst often being perceived as problematic. Youth organisations and youth workers use a variety of means to evaluate and evidence their work. Youth workers and youth work managers generally make active attempts to evaluate youth work in sensitive ways and/or to avoid what they view as inappropriate evaluation. This is supported where practitioners and young people are meaningfully engaged in evaluation processes and/or where funders, managers and decision makers shield youth workers and volunteers from what can be experienced as a bureaucratic burden. 3) Youth workers emphasise that evaluation needs to be flexible, suitable to youth work's distinctive informal education approach, and adaptable across diverse settings and communities. 4) Young people have diverse responses to evaluation, ranging from enthusiastic engagement to overt resistance and opposition, depending on their experiences and perspectives of evaluation in youth work and other settings. Young people prefer processes they perceive as meaningful and/or quick to carry out, and emphasise the need for approaches that are not intrusive, labelling, triggering or stereotyping. Young people tended to be aware that youth work was vulnerable to cuts and that some forms of monitoring and evaluation may help with funding; some took part in evaluation for this reason, and generally felt they were answering evaluation questions accurately, but would prefer not to have to engage in some of these processes. Some young people avoided engaging in evaluation, or said they felt formal evaluation was inappropriate in youth work settings. 5) Young people highly valued open youth work and their relationships with youth workers. They were articulate in expressing how youth work spaces and relationships contributed to the quality of their everyday lives in diverse ways, and how their involvement in youth work had - for many - had a life-changing influence on their lives. 6) Young people, youth workers, managers and policy makers / influencers felt that policy relating to youth work in England could better demonstrate an understanding of the distinctive value of open youth work and more robustly support its development, including by providing more substantial and longer-term funding. |
| Exploitation Route | Academic: The research contributes to scholarship in youth studies, public policy, the sociology of youth, childhood and education, and anthropology, particularly around the governance, provision and evaluation of young people's services. As further journal articles are published from the study, we anticipate the research will continue to contribute to interdisciplinary social science research including through engagement with additional areas including social geography / children's geographies, and we have begun to collaborate with researchers who are researching evaluation, datafication and impact measurement in other contexts and other fields. Non-academic: The research contributes to more participatory and democratic approaches to evaluation and accountability that are suited to the distinctively informal and youth-centred nature of open youth work. It does this by raising awareness of the dangers of inappropriate evaluation mechanisms that risk burdening practitioners, dissuading young people from taking part, and reshaping provision towards short-term projects that are more easily measurable, as well as supporting the development of democratic, inclusive and flexible approaches in policy and practice. The researchers have engaged with youth workers, other youth practitioners, young people, youth and community managers, funders, policy influencers and policy makers throughout the research, aiming to influence policy development and funding requirements, and support the work of practitioners and organisations. More detail is provided in the impact narrative. |
| Sectors | Communities and Social Services/Policy,Education,Leisure Activities, including Sports, Recreation and Tourism,Government, Democracy and Justice |
| URL | https://www.youthandpolicy.org/articles/valuing-and-evaluating-youth-work/ |
| Description | The research has begun to contribute to the effectiveness of public services and policy in the area of youth services, with secondary potential benefits to the quality of life of young people, youth workers and others who work with young people, both in the UK and internationally. There has been significant interest in the study from a variety of stakeholders, including policy makers (civil servants, policy advisors, policy influencers), voluntary and community sector organisations, and research and evaluation companies / social enterprises - in relation to youth work as well as more broadly in the youth, child and education sectors. This impact has been primarily in England, Scotland, Wales and Northern Ireland, but there has also been a growing interest internationally, and the research has been shared with youth practitioners, researchers (academic and non-academic) and policy influencers at events in Ireland, USA, Japan, Malaysia, and Georgia. Through a variety of public engagement activities including workshops, practitioner conferences, training, policy briefings, and bespoke advisory meetings, we have influenced policy, practice and debate, particularly in relation to the following areas: - Sharing and centring the perspectives, experiences and needs of young people and practitioners in relation to youth impact and evaluation. - Understanding the indirect impact of various evaluation and accountability mechanisms on grassroots services, and on the overall provision of services (e.g. there is a danger that 'hard to measure' services such as youth work might lose out if all services need to be measured in the same way). - Taking seriously the need for evaluation and accountability mechanisms that are congruent with practice, and creatively respond to these challenges. The research is contributing to a move towards democratic and youth-centred approaches to evaluation and accountability, that are well suited to the distinctively informal and youth-centred nature of open youth work. It is doing this by (1) raising awareness of the dangers of inappropriate evaluation mechanisms that risk burdening practitioners, dissuading young people from taking part, and reshaping provision towards short-term projects that are more easily measurable, and (2) supporting the development of democratic, inclusive and flexible approaches in policy and practice. --- The research has: --- (a) Influenced policy development and funding Policy development in the youth sector involves government (DCMS), key organisations (notably the Centre for Youth Impact [CYI] set up by the UK's Cabinet Office in 2014, and the National Youth Agency [NYA]), local authorities, and funding bodies. The researchers have engaged with these bodies through a variety of means, including: talks, workshops, blog posts and essays for CYI; two published policy briefings (2019 and 2022); regular informal advice / critical friend discussions with CYI and NYA; informal advice to local authorities and funding bodies; 'Chatham House rules' discussions with DCMS civil servants and Advisors to Ministers and Shadow Ministers; and informing the Labour Party's policy development and manifesto commitment on youth work and its evaluation. In 2021 we were specifically invited to take part in the DCMS / Treasury Review of Government Youth Policies (both through invited meetings with the consultants informing the review, and invited involvement in the National Youth Agency Board's Data and Insights Group). We have hosted roundtable dialogical discussions with youth workers, funding bodies, youth organisation managers, and policy makers and influencers, and brought these groups together to share their views and perspectives, using the research findings as a stimulus. Through these means, the research has contributed to moves towards an understanding in policy of young people's and practitioners' perspectives, leading to more youth-centred approaches that are more in keeping with the philosophy of youth work. --- (b) Supported the work of practitioners and youth organisations: The research has supported youth organisations to evaluate their work in ways that are democratic, youth-centred, and congruent with practice. It has done this through practice-focused events, practice-focused writing and a film, and bespoke engagement with youth organisations (voluntary sector, local authority, social enterprises). The events have included our own practitioner conferences in 2019 and 2021, and events organised in partnership with other organisations in the UK (e.g. delivering training in qualitative research for youth workers with London Youth; speaking at Partnership for Young London and Children and Young People Now events; co-organising a workshop for practitioners, researchers and policy makers working in youth, education and health) and internationally (e.g. speaking at youth work training and education courses in the USA; collaborating with youth workers in the UK and Japan to organise and speak at seminars for over 200 practitioners and local government officers in Japan; speaking at a practice reflection event in Ireland; presenting the research to youth participation professionals in Malaysia). We worked with a youth film company (Mouth That Roars) and three youth work organisations (Mary's Youth Club; Youth Focus North East; Space*) to create a young person led film on the value of youth work, inspired by the research findings. This film formed the basis of three workshops with young people at youth clubs and has been shown at events in the UK and Japan. We have published a practice resource to support practitioners and young people in relation to the evaluation of youth work . ----- We will continue to engage with communities, organisations, young people and policy makers and influencers in relation to the research. Recently we have published a second policy briefing, and a practice reflection and activity resource. Our future plans (in progress) include open access resources for youth and community work trainers and lecturers, further blogs and resources to support our published peer reviewed articles, and resources to support the use of our open access dataset. |
| First Year Of Impact | 2019 |
| Sector | Communities and Social Services/Policy,Education,Leisure Activities, including Sports, Recreation and Tourism,Government, Democracy and Justice |
| Impact Types | Societal,Policy & public services |
| Description | APPG Youth Affairs consultation |
| Geographic Reach | National |
| Policy Influence Type | Contribution to a national consultation/review |
| Description | Citation in APPG Youth Affairs Youth Work Inquiry |
| Geographic Reach | National |
| Policy Influence Type | Citation in other policy documents |
| Impact | The All Party Parliamentary Group on Youth Affairs cited Tania de St Croix's submission (underpinned by this research) to emphasise the significant contributions of youth workers providing young people with trusted relationships, safe spaces and stimulating activities. |
| URL | https://nya.org.uk/appg-inquiry-final-report/ |
| Description | Cited in government-funded youth programme evaluation |
| Geographic Reach | National |
| Policy Influence Type | Citation in other policy documents |
| Impact | Influenced the evaluation of the Youth Investment Fund (government funded programme of open access youth work) in relation to the dangers of conventional outcomes-based monitoring and the importance of youth-centred and practice-sensitive approaches. |
| URL | https://www.thinknpc.org/resource-hub/youth-investment-fund-learning-and-insight-paper-four/ |
| Description | Contributed to LGA Outcomes Framework consultation |
| Geographic Reach | National |
| Policy Influence Type | Contribution to a national consultation/review |
| Impact | Contribution to consultation emphasised importance of overall tone and language; acknowledging space for different views on outcomes; alternatives to outcome based approaches; guidance/acknowledgement re social justice and social change outcomes. Contribution welcomed by email by Bethia McNeil, Director of Centre for Youth Impact, and elements appeared to be included or taken on board. |
| Description | Developed training module 'From anecdote to evidence: Qualitative research skills for youth workers' |
| Geographic Reach | Local/Municipal/Regional |
| Policy Influence Type | Influenced training of practitioners or researchers |
| Impact | Training in qualitative research for youth work practitioners to inform their evaluation and evidence gathering. Participants reported they had increased their knowledge about qualitative research (3.7/5), their confidence in carrying it out (4.1/5) and that the training was useful for their work (4.3/5). |
| Description | Influencing Labour Party policy on youth work |
| Geographic Reach | National |
| Policy Influence Type | Contribution to a national consultation/review |
| Impact | After the researchers contributed to a Labour Party consultation on youth work policy, the Labour Party Shadow Minister for Youth Affairs drew on the research in preparing a significant 2019 Labour Party youth policy paper 'Only Young Once' which committed to reintroducing youth work in all local authorities. |
| URL | https://labour.org.uk/wp-content/uploads/2019/10/Only-Young-Once.pdf |
| Description | Invited contribution to DCMS policy review on youth services |
| Geographic Reach | National |
| Policy Influence Type | Contribution to a national consultation/review |
| Impact | Through invited contributions to DCMS reviews of spending and delivery of youth services, Tania de St Croix emphasised the importance of grassroots open access youth work provision, and of youth-centred and practice-sensitive approaches to evaluation and accountability in youth work. |
| Description | Invited expert for Hope Collective's 'Reimagined' project to create a fairer society for and with young people. |
| Geographic Reach | National |
| Policy Influence Type | Participation in a guidance/advisory committee |
| Description | Membership of National Youth Agency Research Hub Advisory Committee |
| Geographic Reach | National |
| Policy Influence Type | Participation in a guidance/advisory committee |
| Impact | Informing National Youth Agency Research Hub on the potential for academic research and its contribution to the youth sector. |
| Description | Steering Group membership: Setting the research agenda with and for young Londoners |
| Geographic Reach | Local/Municipal/Regional |
| Policy Influence Type | Participation in a guidance/advisory committee |
| Impact | This steering group directs and advises on a project run by the Partnership for Young London, Centre for Youth Impact and London Youth, which aims to create the conditions for the services and support for young people to be: · More informed by research, practice and the experiences of young people themselves; and · More joined up through local and regional collaboration; and thus · More effective This is a two-year funded programme by City Bridge Trust as part of the Cornerstone Fund The partnership aims to generate, mobilise and utilise knowledge and insight to change the life chances of young people and aims: 1. To improve services for young people in London through better use of data and research 2. To increase coordination and reduce duplication of effort on data gathering and research 3. To facilitate closer links between research and practice, increasing practitioners use of research and researchers impact on practice It aims to achieve this by: · Creating a stronger consensus and shared commitment to the potential for a collaborative research agenda to contribute to improved life chances for young people in London · Developing a clearer understanding of the roles of key stakeholders in making a collaborative research agenda a reality · Taking forward the collaboration on the Vision for Young Londoners with, and for young people As part of this agenda, Tania de St Croix's involvement in the steering group has contributed expertise in bringing academic research and practice closer together in the area of youth services, as well as recommendations on including a focus on open / universal youth work and its appropriate evaluation. |
| Description | Submitted evidence to statutory guidance review for local youth services (DCMS) |
| Geographic Reach | National |
| Policy Influence Type | Contribution to a national consultation/review |
| URL | https://www.gov.uk/government/consultations/statutory-guidance-review-for-local-youth-services-have-... |
| Description | Use of research-derived reflection questions in evaluation database application brochure |
| Geographic Reach | Local/Municipal/Regional |
| Policy Influence Type | Contribution to new or Improved professional practice |
| Impact | Teckle Data is marketed as 'an online evaluation database tool built by a grass roots charity, for grass roots charities' and aims to enable and resource ethical and practice-sensitive approaches to collecting information. While our questions are only one part of this brochure, most of which is the work of Hot Chocolate, our ongoing relationship with Hot Chocolate is an example of our bespoke collaborative 'critical friend' relationships with many youth, community and evaluation organisations and professionals who are working towards practice-sensitive approaches to evaluation, and who have told us that our research is useful in developing their thinking and practice on these issues. |
| Title | Rethinking Impact, Evaluation and Accountability in Youth Work, 2018-2021 |
| Description | The data collection included 87 qualitative interviews / focus groups (semi-structured and/or flexible / conversational / creative) with 143 people: 58 young people who engage in youth work, 59 youth workers and managers, and 26 policy makers, influencers and informants. The data focuses primarily on the value of youth work and its evaluation. The young people, youth workers and managers were from eight open youth work settings around England, selected to represent the diversity of open youth work (youth clubs in purpose built centres and shared spaces; detached / street based youth work; and open youth work with specific groups: trans young people, girls, and boys). The policy makers, influencers and informants were mostly from England. One was from Scotland and five from the USA; these perspectives were sought for comparative and international learning purposes. In addition, the researchers engaged in 73 sessions of participant observation. 63 of these were in the eight youth work settings mentioned above. The remaining ten were policy-related events. The fieldnotes and a small number of interview / focus group transcripts are not included in the shared dataset, either for ethical reasons (e.g. if it was not feasible to anonymise them or to redact sensitive data), or because participants opted out of data sharing. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2021 |
| Provided To Others? | Yes |
| Impact | To date there have been 6 data downloads and 39 page views since the dataset was published. We plan to share the dataset through open-access training resources produced for and with youth work trainers and lecturers, and through links from our publications. |
| URL | http://reshare.ukdataservice.ac.uk/id/eprint/855316 |
| Description | Article for The Story practitioner magazine |
| Form Of Engagement Activity | A magazine, newsletter or online publication |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | Invited by YouthScape to write article on the research for 'The Story' |
| Year(s) Of Engagement Activity | 2018 |
| URL | https://www.youthscape.co.uk/research/publications/the-story-7 |
| Description | CYPN article |
| Form Of Engagement Activity | A magazine, newsletter or online publication |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | Invited to write short article for a special section in main youth sector magazine, Children and Young People Now. Study was also featured in the same issue as one of four research summaries. This led to enquires and interest from practitioners and managers in the third sector, public sector and social enterprises in England and Scotland, and probably to the CYPN conference presentation reported separately. |
| Year(s) Of Engagement Activity | 2018 |
| URL | https://www.cypnow.co.uk/cyp/feature/2005588/special-report-youth-work-impact |
| Description | Cited as research having an impact |
| Form Of Engagement Activity | A magazine, newsletter or online publication |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | Cited in TAG Improving Research and Impact in Youth and Community Work 'Is youth and community work research having an impact?' as an example of research having impact. |
| Year(s) Of Engagement Activity | 2020 |
| URL | https://www.tagpalycw.org/newsblog/2020/5/24/members-connect-40-research-and-impact-post-covid19 |
| Description | Collaboration with Red Wheelbarrow, Ireland |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Contributed to organised discussion in January 2021 for youth workers in Ireland on themes of evaluation, accountability and neoliberalism in youth work. This was followed up by an invitation to participate in a national conference in June 2021 for practitioners in Ireland and beyond, 'What's the story - narrative in youth work and youth studies', in which we contributed a short research presentation as part of a wider collaboratively planned session including a performed dialogue on the theme 'is governance good for you?' |
| Year(s) Of Engagement Activity | 2021 |
| Description | Deliberative workshops with practitioners, managers, policy makers and influencers, funders |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | Four deliberative workshops of two hours each enabled in-depth discussion of and reflection on the themes and findings of the research. Externally facilitated by Graeme Tiffany using community philosophy methodologies, the four workshops were aimed respectively at: frontline youth workers; youth work managers; policy makers and influencers; funders. A final deliberative workshop for a large group brought the concerns and ideas of all four groups together. |
| Year(s) Of Engagement Activity | 2021 |
| Description | Environmental education blog |
| Form Of Engagement Activity | Engagement focused website, blog or social media channel |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | As a result of attending a research seminar on our research, Melissa Glacken wrote this blog for the National Association of Environmental Education based on our research and its relevance for their sector, saying "their insights resonate with policy formation and practice within the environmental education and outdoor learning sectors". |
| Year(s) Of Engagement Activity | 2019 |
| URL | http://naee.org.uk/impact-agenda-influence-practice/ |
| Description | Featured in Play It Forward e-newsletter (USA) |
| Form Of Engagement Activity | A magazine, newsletter or online publication |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Tania de St Croix was interviewed for the Play It Forward e-newsletter, resulting in an article on the research in the series 'a new conversation about evaluation' (USA). |
| Year(s) Of Engagement Activity | 2021 |
| URL | https://allstars.org/play-it-forward/ |
| Description | How can we tell if it's good without making it worse? Research, practice and policy in dialogue (CPPR workshop) |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | Event organised as part of the research project, in collaboration with fellow researchers and practice and policy partners as a Centre for Public Policy Research workshop / symposium to bring together practitioners, policy makers and researchers in education, youth work and health. Tania spoke about the Rethinking Impact research as part of the workshop. Event description: Evaluation and accountability mechanisms not only measure or assess practice; they also tend to change or reshape that practice, whether intentionally or unintentionally. This symposium, organised by researchers at the Centre for Public Policy Research (CPPR) at King's College London, aims to bring together researchers, practitioners and policy makers from a range of settings (including education, youth work and health) to discuss the relationship between the 'substance' of practice - what makes it 'successful' or 'good', what 'works' and what 'counts' - and what processes or methodologies might be used to evaluate this success. We will come together to discuss common themes and questions, sparked by a small number of short (3-5 minute) provocations from a range of perspectives. We hope that this symposium will be a space for generative and thoughtful conversations, and to share commonalities, similarities, differences and insights across different practice settings and research interests in relation to these themes. |
| Year(s) Of Engagement Activity | 2022 |
| Description | London Regional Impact Network |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Professional Practitioners |
| Results and Impact | Tania participates in the London Regional Impact Network (linked to Centre for Youth Impact and London Youth), feeds into the discussions and consultations that happen through this group, and was asked to present an example of an evaluation tool (story-telling in youth work) at a meeting on 17th July. |
| Year(s) Of Engagement Activity | 2018,2019 |
| Description | London Youth Research Network |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Professional Practitioners |
| Results and Impact | London Youth Research Network set up by Partnership for Young London, London Youth, Centre for Youth Impact and others. Tania was asked to speak at a launch event for the group and invited to become part of the steering group. Involvement includes contributing ideas to the group, emphasising the need to listen to young people and practitioners' perspectives, make links between university research with practice-based research, and understand research widely (rather than narrowly as impact measurement). |
| Year(s) Of Engagement Activity | 2018,2019 |
| Description | Online workshops with youth workers and others in USA |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Postgraduate students |
| Results and Impact | Three workshops / talks at postgraduate youth and community work training in University of Madison, Wisconsin, and Harvard University, USA. |
| Year(s) Of Engagement Activity | 2020,2021,2022 |
| Description | Online workshops with youth workers in Japan |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Two online workshops organised by a group of youth workers and youth work researchers in Japan. At the second, Tania presented the Rethinking Impact research. The group had engaged with this research through watching published presentations and the associated film, and we discussed what forms of evaluation could be most appropriate for youth work, and agreed to continue this discussion at a later date. The group found the critiques of dominant evaluation approaches, and young people's and youth workers' views, was relevant in their context as youth workers in Japan. |
| Year(s) Of Engagement Activity | 2021,2022 |
| Description | Podcast - impact, evaluation and accountability in youth work |
| Form Of Engagement Activity | A broadcast e.g. TV/radio/film/podcast (other than news/press) |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | In this podcast a panel of researchers from three countries - Tania de St Croix (Kings College London), Sinead McMahon (Limerick Institute of Technology, Ireland), Bianca Baldridge (University of Madison Wisconsin, USA) and Louise Doherty (Kings College London) - discuss impact, evaluation and accountability in youth work, thinking critically about the application of impact in youth work. The group explores the implications of the rise of economistic and quantitative measures of the value of youth work in policy and research, and the effects on youth work practice and its ability to address the inequities young people face. The podcast was organised by and is hosted by the British Educational Research Association and resulted from a symposium on the same theme brought together by the Rethinking Impact study. |
| Year(s) Of Engagement Activity | 2021 |
| URL | https://www.bera.ac.uk/media/youth-and-informal-education-podcast-discussing-impact-evaluation-and-a... |
| Description | Practitioner event 'Thinking critically about impact, evaluation and accountability in youth work' |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | 45 people took part in a full day event in which we shared our early findings from the research. |
| Year(s) Of Engagement Activity | 2019 |
| URL | https://www.kcl.ac.uk/news/sell-out-youth-work-event-energises-practitioners |
| Description | Presentation (Georgia) |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Postgraduate students |
| Results and Impact | Talk entitled "Youth work: informal education, evaluation and democracy" for an event entitled 'Democracy and Education' on Unesco's International Day of Education at the International Relations and Development Department, Georgian Institute of Public Affairs, Georgia. Followed by discussion and questions. |
| Year(s) Of Engagement Activity | 2022 |
| Description | Presentation at International Webinar on Youth-Adult Co-Creation |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Talk entitled 'Co-creation with young people in practice and research: Building on youth work practice', including findings from the research on co-creating evaluation with young people, at international event based in Malaysia and including talks from Hong Kong and Taiwan. |
| Year(s) Of Engagement Activity | 2021 |
| Description | Presentation at Lancaster University |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | Presentation at Lancaster University by Tania, drawing together previous and current study, with around 15 practitioners, researchers and students in attendance, and many more registered for the live stream - many of these took part in the discussion, including a group of students on a professional qualifying course, and youth workers from Scotland. Sparked engaged conversation, reported by hosts as the most engagement of any talk at the centre so far. |
| Year(s) Of Engagement Activity | 2018 |
| Description | Project website and email list |
| Form Of Engagement Activity | Engagement focused website, blog or social media channel |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Engagement-focused website and email list. Over 100 on email list by February 2019. |
| Year(s) Of Engagement Activity | 2018,2019 |
| URL | http://rethinkingimpact.com |
| Description | Question to Minister sparked debate |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Policymakers/politicians |
| Results and Impact | Tania de St Croix was invited to pose a question to the Minister for Civil Society at a National Youth Agency policy event. The question concerned the long-term funding of year-round open access youth work and the delay to the Youth Investment Fund. The Minister's answer sparked debate on social media, and a blog piece. |
| Year(s) Of Engagement Activity | 2021 |
| URL | https://youthworkable.wordpress.com/2021/02/12/inno-fucking-vation/ |
| Description | Quoted in magazine article |
| Form Of Engagement Activity | A press release, press conference or response to a media enquiry/interview |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Public/other audiences |
| Results and Impact | Interviewed by Ciaran Thaper and quoted in article on youth work cuts in Thinking City magazine |
| Year(s) Of Engagement Activity | 2018 |
| URL | https://thinkingcity.org/2018/10/09/inside-the-decline-of-londons-youth-clubs/ |
| Description | Rethinking Impact conference (end of project research and practice conference) |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | 80 youth workers and other interested people attended and participated in a one-day online conference in which we presented the research, engaged in deliberative discussion, and engaged in workshops run by project partners on themes arising from the research. Feedback was highly positive and participants suggested they had reflected on the themes and developed their thinking and practice on evaluation in youth work settings. Two participants wrote a follow-up piece reflecting on their involvement in the conference: https://sites.google.com/view/youth-work-dialogue-ni/critical-voice-articles and https://docs.google.com/document/d/16qxJctfutTSdXegjuvkthUnlrUp5Etvq/edit . See also feedback on twitter using hashtag #RethinkingImpact |
| Year(s) Of Engagement Activity | 2021 |
| URL | https://www.kcl.ac.uk/events/rethinking-impact-conference |
| Description | Speaker at events in Wales |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | Invited speaker at three events in Wales to inform debate on evaluation in relation to voluntary services and/or youth services. Wales Centre for Voluntary Services event, 'People are not data points', 2020 WCVA Inspiring impact event, 'Quality of evidence v quality of experience', 2020 Youth Sector conference, October 2021 - keynote provocation looking at the Wavehill report (commissioned by Welsh government) and especially the elements around evaluation |
| Year(s) Of Engagement Activity | 2020,2021 |
| Description | Speaking at events in London on youth work and its impact |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Professional Practitioners |
| Results and Impact | Speaking at events in London on youth work and its impact e.g. Partnership for Young London event on youth work, research and evidence Partnership for Young London consultation and launch of report on the impact of open access youth work London Youth panel on youth work and research |
| Year(s) Of Engagement Activity | 2019,2020,2021 |
| Description | Story-telling in youth work - workshops |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Story-telling workshops facilitated by Tania and/or Louise collaborating with In Defence of Youth Work: 2018 At Czech street workers conference (12 attendees) At Plymouth Transforming Youth Work conference - sharing with international youth work networks, co-facilitated with Japanese youth workers and researchers (30-40 attendees) |
| Year(s) Of Engagement Activity | 2018,2019 |
| URL | http://story-tellinginyouthwork.com |
| Description | Supporting evidence review of open access youth work |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | Tania de St Croix worked with Partnership for Young London, Centre for Youth Impact and London Youth to plan and consult on an evidence review of the impact of youth work on young people. She recruited and mentored the researcher and took part in producing the review and its launch event. |
| Year(s) Of Engagement Activity | 2020 |
| URL | https://www.partnershipforyounglondon.org.uk/post/open-access-youth-work-a-narrative-review-of-impac... |
| Description | Talk at CYPN practitioner conference |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | Invited to give workshop on 'rethinking impact' - organiser said it was the most popular workshop at the conference. People said they came because either they are worried about the impact of impact, or work in impact and want to understand the critique. Received excellent feedback on Twitter, e.g. "Helping us consider critiquing the grounds on which impact has become such a dominant force in the sector. Fascinating insight on how evaluation-done-bad can negatively change the relationships on the front line in #youthwork.", "It was like a breath of fresh air to hear her speak", "Dr de St Croix is a youth worker warrior and bloody love it!" |
| Year(s) Of Engagement Activity | 2018 |
| Description | Transforming Youth Work conference |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Talk and workshop on our research at international Transforming Youth Work conference. 30-40 people attended from several countries, and engaged in useful debate. |
| Year(s) Of Engagement Activity | 2018 |
| Description | UNCRC workshop with young people |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Schools |
| Results and Impact | 10 young people took part in a workshop in which we drew on our research to engage in creative activities and debate whether young people should have a 'right' to youth work. |
| Year(s) Of Engagement Activity | 2019 |
| URL | https://www.ucl.ac.uk/ioe/events/2019/nov/celebrating-childrens-rights-ucl |
| Description | Young people's film: The value of youth work |
| Form Of Engagement Activity | A broadcast e.g. TV/radio/film/podcast (other than news/press) |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Public/other audiences |
| Results and Impact | 'The value of youth work' is a film made by young people from Mary's Youth Club, Space Youth Services (Devon) and Youth Focus North East, working with youth work film company Mouth That Roars and inspired by the themes and findings of the research. This is a publicly available ten-minute film that demonstrates clearly young people's views on the value of youth work. It has been widely shared and is aimed at decision makers, funding bodies, and anyone who wants to understand the contribution of youth work, and/or use it for discussions or training with young people, colleagues, and/or students. |
| Year(s) Of Engagement Activity | 2021 |
| URL | https://vimeo.com/597256970 |
| Description | Youth Work in Japan |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Tania de St Croix alongside a youth work professional Colin Brent, a senior youth worker from Ealing Council, were invited to Japan to conduct workshops with youth workers, researchers, students and local government administrators in three cities: Sapporo, Tokyo and Kyoto. They shared their research, experiences and perspectives of youth work in the UK, as well as discussing their work as part of In Defence of Youth Work, which aims to defend critical emancipatory youth work in the English context and build international solidarity amongst youth workers. Tania and Colin were invited by Maki Hiratsuka, a Professor at Hosei University in Tokyo, and her research collaborators who are researching youth work in Europe in order to develop the practice of youth work in Japan. They are particularly interested in 'youth work stories' - as part of In Defence of Youth Work, Tania and Colin (along with others) have developed storytelling workshops that enable youth workers to evaluate and reflect on their work in all its complexity. This relates to the study Tania and Louise Doherty are undertaking, which argues that evaluation of youth work needs to take into account its social context, and be developed in democratic and practice-relevant ways. The Japanese youth workers and researchers have adapted the storytelling method into what they call a 'counter documentation movement' and published an e-journal 'How do we organise youth work?' where they include Japanese youth work stories in order to progress the development of youth work and youth workers in the Japanese context. |
| Year(s) Of Engagement Activity | 2019 |
| URL | https://www.kcl.ac.uk/news/a-trip-to-japan-youth-work-research-goes-global |