Meaningful and measurable: developing approaches to the analysis and use of information on personal outcomes within health and social care

Lead Research Organisation: University of Edinburgh
Department Name: Sch of Health in Social Science

Abstract

Policy and practice across the UK and beyond is committed to improving the lives of people who use services and unpaid carers. Over the past five to ten years there has been a particular concern to shift health and social care systems away from an exclusive focus on their own inputs, processes and outputs to give a more prominent focus to personal outcomes for people using services and unpaid carers. In this context the term personal outcomes is used to refer to both the impact and end results of services and supports on a persons life and more simply, what matters to people.

The proposed project builds on a seven year programme of knowledge exchange and service improvement that has been led by two of the applicants (Dr Ailsa Cook and Dr Emma Miller) and funded by the Joint Improvement Team in Scotland. This programme, known as Talking Points, has involved work with more than 130 organisations to support the development of outcomes focussed practice. A key finding emerging from the Talking Points programme is that limited capacity and skills in the analysis of qualitative and quantitative personal outcomes information within health and social care organisations constitutes a significant barrier to effective outcomes focussed project. Furthermore, this issue is exacerbated by the predominance of performance cultures that prioritise consistency, comparability and measurability of information over meaning.

Over the past year 15 partner organisations have worked together to develop this proposal for funding from the ESRC. The partner organisations have all been involved in the Talking Points programme and include the following academic, practice and national stakeholder partners:
University of Edinburgh (host institution), University of Strathclyde, University of Swansea
Angus Council, Bridgend County Borough Council, City of Edinburgh Council, East renfrewshire Health and Care Partnership, Moray Health and Social Care Partnership, Penumbra, Stirling Council, VOCAL (Voice of Carers Across Lothian)
Joint Improvement Team, Community Care Benchmarking Network, The ALLIANCE, Social Services Inspection Agency.

The proposed project involves both collaborative action research and knowledge exchange elements. Specifically the academic team will support each of the practice partners to engage in an action research project focussed on the analysis and use of personal outcomes data routinely collected through assessment and review processes within their organisations. Co-ordination and collaboration between these projects will be primarily achieved through the convention of three 'data retreats' two day workshops where project partners work together to develop capacity and skills, exchange knowledge and reflect on the process.

The contribution of national stakeholder partners will ensure that the findings of this project are effectively disseminated to and implemented within a wider constituency of health and social care organisations. This will be achieved through support for a range of knowledge exchange processes, including events, dissemination of materials and staff time to work with other organisations not directly involved in the project to implement the findings in their own practice. Engagement with these partners, all of whom are actively involved in shaping policy, will ensure that the project findings influence national policy, in particular in relation to the development of national outcome performance reporting frameworks.

In these ways it is anticipated that the proposed project will bring significant benefits not only to partner and provider organisations, but to the wider health and social care system. The short term impacts of the project will be systematically evaluated through a process of contribution analysis. Findings from this contribution analysis and the project as a whole will be widely disseminated, including to academic audiences.

Planned Impact

The proposed project builds on a seven year programme of service improvement and action research led by two of the applicants (AC and EM) and funded by one of the stakeholder partners (the Joint Improvement Team) that has involved work with more than 130 organisations in Scotland and Wales, including all of the project partners. This programme of work has transformed the policy landscape in Scotland with personal outcomes emphasised in eight major policy strategies since 2008. A recent survey of health and social care partnerships in Scotland found that 28/32 partnerships are implementing personal outcomes focussed approaches with 26 of these using the 'Talking Points: Personal Outcomes Approach' developed by the applicants (Scottish Community Care Benchmarking Network, 2013).

Against this backcloth the project team are confident that the proposed project will have significant impact on practice within the partner organisations and wider policy and practice contexts in which they exist. By seeking to change organisational practice in the short term the project has the capacity to impact longer term on staff within those organisations and people using services and their unpaid carers. In so doing the project contributes to the ESRC strategic aims of improving health and wellbeing of the population, engaging the public and improving the effectiveness of public services.

Specifically we anticipate the following short term impacts for practice partner organisations:

1. Enhanced collation, analysis and reporting of routinely collected information on personal outcomes.
2. Practitioners, managers and commissioners will use reports based on personal outcomes information to inform staff development, service improvement and the planning and commissioning of services and support.
3. Strategic planners and managers will use reports based on personal outcomes information to consider organisational performance in relation to outcomes for individuals alongside information on inputs, processes and outputs. This information will be used in decision making by Chief Executives and Boards and as evidence for inspection and regulation.

The longer term impacts include:
1. The presence of credible and robust information on personal outcomes routinely reported to Strategic managers and Boards will support the transformation of organisations to more outcomes focussed ways of working.
2. People using services and unpaid carers will benefit from more personalised and outcomes focussed engagement with practitioners, leading to more effective support plans and the associated benefits to quality of life.

The contribution of national stakeholder partners to the project ensures these impacts may also be realised by health and social care organisations working to implement outcomes focussed approaches not included in the project. The engagement of national stakeholders, all of whom have a critical role in shaping as well as implementing policy, will ensure that the project findings influence wider policy debates. The findings are of particular relevance to the development and implementation of national outcomes performance frameworks in Scotland, England and Wales. By capturing evidence of the benefits and challenges of focussing on meaning and measurement within improvement and performance processes the project has the potential to influence policy at a national level, as well as academic debates in this area.

Achievement of these impacts is contingent on a number of factors external to this project. A risk to the project delivering on these impacts is that the current policy focus on personalisation and outcomes is overtaken by a new policy agenda. Review of the relevant policy strategies as well as the investment in the project by the partners leads us to be confident that the issues addressed in this project are likely to become more critical and prominent over time and not less.

Publications

10 25 50
 
Description Measurement of personal outcomes:
The extent to which it is it feasible to 'measure' personal outcomes is contested and subject to varying conceptualisations, which emerged as barriers to progress. In response and through collective exploration, the academic team produced briefings, since incorporated as a practical guide:
o The pressure to measure and its impact on understandings about qualitative data
o The need to reconcile different conceptualisations of personal outcomes
o Thinking about categorisation of outcomes and avoiding the risk of categorizing too soon
o Choice of scale measure in accordance with the conceptualisation of personal outcomes, with practice, and acceptability to people using the service

An agreed conclusion was that quantitative data about personal outcomes needs to be used with caveats and not in isolation from qualitative and contextual information.

Recording personal outcomes:
The importance of focusing on practitioner recording skills to generate good quality information and decision-making at individual and collective levels, was already known. However, as this issue re-emerged as a concern for all eight project partners, and featured in all of their action research projects, efforts were invested in exploring the epistemological and practical barriers involved, and in developing common criteria to support good quality narrative recording. This work has been published as a standalone practice guide, incorporating examples which cover a range of barriers identified by practitioners and representing health, social work and third sectors.

Use of qualitative data in practice settings:
Recognising the limits of quantitative outcomes data, most project partners viewed qualitative data as important for decision-making. It was however apparent that understandings of qualitative data remain limited in service settings. First, there is a widespread tendency to equate 'qualitative data' with 'story', resulting in potential misuse of individual stories in informing decision-making. Second, when working with qualitative data:
• Quantitative data sampling assumptions and criteria are carried forward, resulting in inappropriate use of large, 'random' samples;
• Uncertainty about the conditions under which qualitative findings might be applied more broadly results in myths about 'how many' cases are needed, and a lack of attention to selection criteria.
The researchers worked with partners to progress qualitative data analysis in service settings, resulting in briefings addressing the above themes and an overview of different approaches. The resulting guide presents a novel approach through tackling common misunderstandings in service settings.
Use of data about personal outcomes
It was already known that significant effort must be invested in recording practice to generate good quality data. This project confirmed the importance of the feedback loop within the organisation, in using personal outcomes data for practice development purposes, with partners identifying methods for doing this. Importantly, collaborative and appreciative approaches were found to be more effective, in contrast to the more prescriptive, top down approaches currently employed for data capture. While subject to continuing analysis, implications emerge for how the data in turn inform bottom up approaches to performance management, with novel ideas emerging about links to service planning and commissioning, representing a rich seam for exploration through future research.
Exploitation Route The academic team is committed to producing high quality journal articles, to develop the currently limited evidence base around personal outcomes, with four articles planned. A team member has already fed the findings into the kick-off meeting of the eurocarers research working group in Dublin in April 2015 which will influence H2020 themes on carers, with a related paper accepted at the International Carer Conference in Gothenberg in September. Formal evidence has been identified as important to policy and practice colleagues, especially those working in healthcare settings, in being able to continue to promote and embed the work, in Scotland, Wales and beyond.

Our third and final knowledge exchange event involved 60 multi-sectoral stakeholders from Scotland, Wales and England, many with direct responsibility for policy, practice and regulation. Feedback emphasised learning about the importance of supporting practitioners and culture change and greater clarity about how these could be achieved. A commitment to promoting the use of qualitative data was a strong theme, which in turn linked to two other priorities, progress with outcomes and performance, and outcomes and commissioning. The findings are being posted on the Meaningful and Measurable website and will permeate the already existing Personal Outcomes Collaboration website which is aimed at policy, practice and academics.
Sectors Communities and Social Services/Policy,Healthcare,Government, Democracy and Justice

URL https://meaningfulandmeasurable.wordpress.com/
 
Description Within Scottish organisations the findings have already been fed directly by the research team into service improvement work in Dumfries and Galloway NHS, Greater Glasgow and Clyde NHS, Shetland Social Services, and with The Richmond Fellowship Scotland senior management team. This has included learning from the project about practical implementation with the three components of outcomes approach, including engagement, recording and use of information. The conclusion of the project has coincided with a new collaboration between the research team and the eurocarers research working group. Two meetings with this group in April and July 2015 are to be followed by a further meeting and presentation at the International Carer Conference in Gothenberg in September. Although not a direct result of the project, the research findings will thus link into a developing European research strategy for carers and potentially the future direction of H2020 priorities relating to carers. Two members of the team are leading a workshop for 30 practitioners on using qualitative data in NHS settings in May/June 2015. These are practitioners who have collected qualitative data in their service improvement work, but don't know how to use it. The project findings will help to achieve greater impact because of new insights into common assumptions and misunderstandings in service settings about use of qualitative data. This is already generating interest from other sectors. At the national level: in Scotland the findings have already fed directly into test work relating to the dementia strategy and related outputs, including aspects of the findings about recording outcomes emerging in national standards for care planning. In England one of the research team fed the findings into a seminar for local authorities in March 2015 prior to piloting work with 12 local authorities. The findings have a strong impact on the direction of the development and future implementation of the national outcomes framework in Wales, as evidenced in the Bridgend project report and the report on using information which is due for completion by July 2015. The lead for outcomes in the Scottish Government aims to link the findings into emerging guidance on the new national outcomes for health and social care integration.
First Year Of Impact 2014
Sector Communities and Social Services/Policy,Healthcare,Government, Democracy and Justice
Impact Types Policy & public services

 
Description Eurocarers Research Working Group research membership and contribution to EU carers research priorities document
Geographic Reach Europe 
Policy Influence Type Membership of a guideline committee
URL https://www.eurocarers.org/Eurocarers-Research-Priorities
 
Description input to Welsh Assembly work on developing an outcomes approach for social services (via skype into national committee and teleconferences)
Geographic Reach National 
Policy Influence Type Influenced training of practitioners or researchers
Impact Members of the Social Services Improvement Agency (SSIA) attended all three of the knowledge exchange events and the final dissemination event of the ESRC project, taking the emerging findings immediately back to their developing approach in Wales. Changed approaches to practice in engaging with members of the public requiring social service support.
 
Description Development grant
Amount £8,000 (GBP)
Organisation My Home Life 
Sector Charity/Non Profit
Country United Kingdom
Start 06/2015 
End 03/2016
 
Description EU H2020 RISE fund
Amount £602,333 (GBP)
Organisation European Commission 
Sector Public
Country European Union (EU)
Start 01/2016 
End 12/2019
 
Description Scottish Government research implementation grant
Amount £17,500 (GBP)
Organisation Government of Scotland 
Department Joint Improvement Team
Sector Public
Country United Kingdom
Start 04/2015 
End 03/2016
 
Title Residential data retreat 
Description The data retreat was central to the collaborative knowledge exchange approach of this project and the dialogical exchanges had direct impact on all of the case study sites, providing opportunities to learn from each other's mistakes and successes 
Type Of Material Model of mechanisms or symptoms - human 
Provided To Others? No  
Impact We are currently writing up a series of journal articles from the project for submission 2016, to include one on the role of the data retreat 
 
Description Personal outcomes network 
Organisation Government of Scotland
Department Joint Improvement Team
Country United Kingdom 
Sector Public 
PI Contribution This network has brought together national health and social care bodies with the two Scottish Universities involved in the ESRC project to form a co-ordinating group. The contributions from the research team have been mainly in time to plan and attend core group meetings and in feeding in findings and outputs from the research
Collaborator Contribution The partners in the co-ordinating group contribute their time for meetings and planning and also significant administrative support to set up quarterly meetings with a wider group of partners including Health Improvement Scotland, Health board representatives and local authority and third sector organisations. The group is interested in continuing implementation of a personal outcomes approach, based on the evidence available
Impact So far the collaboration has resulted in a new set of principles and values underpinning a personal outcomes approach. This has just been signed off and will be promoted through agency websites and distribution lists. It is informed by the research findings and seeks to promote a consistent understanding as well as set of principles for implementation. The network has also had a demonstration of the collaboration website which contains materials to support implementation. Disciplines include health, social work including adult and childrens services, policy
Start Year 2015
 
Description Personal outcomes network 
Organisation NHS Scotland
Country United Kingdom 
Sector Public 
PI Contribution This network has brought together national health and social care bodies with the two Scottish Universities involved in the ESRC project to form a co-ordinating group. The contributions from the research team have been mainly in time to plan and attend core group meetings and in feeding in findings and outputs from the research
Collaborator Contribution The partners in the co-ordinating group contribute their time for meetings and planning and also significant administrative support to set up quarterly meetings with a wider group of partners including Health Improvement Scotland, Health board representatives and local authority and third sector organisations. The group is interested in continuing implementation of a personal outcomes approach, based on the evidence available
Impact So far the collaboration has resulted in a new set of principles and values underpinning a personal outcomes approach. This has just been signed off and will be promoted through agency websites and distribution lists. It is informed by the research findings and seeks to promote a consistent understanding as well as set of principles for implementation. The network has also had a demonstration of the collaboration website which contains materials to support implementation. Disciplines include health, social work including adult and childrens services, policy
Start Year 2015
 
Description Personal outcomes network 
Organisation Scottish Social Services Council
Country United Kingdom 
Sector Public 
PI Contribution This network has brought together national health and social care bodies with the two Scottish Universities involved in the ESRC project to form a co-ordinating group. The contributions from the research team have been mainly in time to plan and attend core group meetings and in feeding in findings and outputs from the research
Collaborator Contribution The partners in the co-ordinating group contribute their time for meetings and planning and also significant administrative support to set up quarterly meetings with a wider group of partners including Health Improvement Scotland, Health board representatives and local authority and third sector organisations. The group is interested in continuing implementation of a personal outcomes approach, based on the evidence available
Impact So far the collaboration has resulted in a new set of principles and values underpinning a personal outcomes approach. This has just been signed off and will be promoted through agency websites and distribution lists. It is informed by the research findings and seeks to promote a consistent understanding as well as set of principles for implementation. The network has also had a demonstration of the collaboration website which contains materials to support implementation. Disciplines include health, social work including adult and childrens services, policy
Start Year 2015
 
Description Personal outcomes network 
Organisation University of Edinburgh
Country United Kingdom 
Sector Academic/University 
PI Contribution This network has brought together national health and social care bodies with the two Scottish Universities involved in the ESRC project to form a co-ordinating group. The contributions from the research team have been mainly in time to plan and attend core group meetings and in feeding in findings and outputs from the research
Collaborator Contribution The partners in the co-ordinating group contribute their time for meetings and planning and also significant administrative support to set up quarterly meetings with a wider group of partners including Health Improvement Scotland, Health board representatives and local authority and third sector organisations. The group is interested in continuing implementation of a personal outcomes approach, based on the evidence available
Impact So far the collaboration has resulted in a new set of principles and values underpinning a personal outcomes approach. This has just been signed off and will be promoted through agency websites and distribution lists. It is informed by the research findings and seeks to promote a consistent understanding as well as set of principles for implementation. The network has also had a demonstration of the collaboration website which contains materials to support implementation. Disciplines include health, social work including adult and childrens services, policy
Start Year 2015
 
Description Personal outcomes network 
Organisation University of Strathclyde
Country United Kingdom 
Sector Academic/University 
PI Contribution This network has brought together national health and social care bodies with the two Scottish Universities involved in the ESRC project to form a co-ordinating group. The contributions from the research team have been mainly in time to plan and attend core group meetings and in feeding in findings and outputs from the research
Collaborator Contribution The partners in the co-ordinating group contribute their time for meetings and planning and also significant administrative support to set up quarterly meetings with a wider group of partners including Health Improvement Scotland, Health board representatives and local authority and third sector organisations. The group is interested in continuing implementation of a personal outcomes approach, based on the evidence available
Impact So far the collaboration has resulted in a new set of principles and values underpinning a personal outcomes approach. This has just been signed off and will be promoted through agency websites and distribution lists. It is informed by the research findings and seeks to promote a consistent understanding as well as set of principles for implementation. The network has also had a demonstration of the collaboration website which contains materials to support implementation. Disciplines include health, social work including adult and childrens services, policy
Start Year 2015
 
Description secure care outcomes research group 
Organisation Centre for Excellence for looked after children in Scotland (CELCIS)
Country United Kingdom 
Sector Academic/University 
PI Contribution This research group is collaborating to develop a research bid based on various areas of expertise relevant to secure care. The research team is contributing both previous experience of supporting secure care centres in Scotland to develop their outcomes approaches, and also the findings of the ESRC project
Collaborator Contribution Each partner has considerable knowledge/expertise in this area and direct access to policy and practice networks
Impact The group is steadily progressing with aims, objectives and theoretical perspectives influenced by the completed research described here.
Start Year 2015
 
Description secure care outcomes research group 
Organisation Centre for Youth and Criminal Justice (CYCJ)
Country United Kingdom 
Sector Charity/Non Profit 
PI Contribution This research group is collaborating to develop a research bid based on various areas of expertise relevant to secure care. The research team is contributing both previous experience of supporting secure care centres in Scotland to develop their outcomes approaches, and also the findings of the ESRC project
Collaborator Contribution Each partner has considerable knowledge/expertise in this area and direct access to policy and practice networks
Impact The group is steadily progressing with aims, objectives and theoretical perspectives influenced by the completed research described here.
Start Year 2015
 
Description secure care outcomes research group 
Organisation Centre for Youth and Criminal Justice (CYCJ)
Country United Kingdom 
Sector Charity/Non Profit 
PI Contribution This research group is collaborating to develop a research bid based on various areas of expertise relevant to secure care. The research team is contributing both previous experience of supporting secure care centres in Scotland to develop their outcomes approaches, and also the findings of the ESRC project
Collaborator Contribution Each partner has considerable knowledge/expertise in this area and direct access to policy and practice networks
Impact The group is steadily progressing with aims, objectives and theoretical perspectives influenced by the completed research described here.
Start Year 2015
 
Description secure care outcomes research group 
Organisation University of Strathclyde
Department School of Social Work & Social Policy
Country United Kingdom 
Sector Academic/University 
PI Contribution This research group is collaborating to develop a research bid based on various areas of expertise relevant to secure care. The research team is contributing both previous experience of supporting secure care centres in Scotland to develop their outcomes approaches, and also the findings of the ESRC project
Collaborator Contribution Each partner has considerable knowledge/expertise in this area and direct access to policy and practice networks
Impact The group is steadily progressing with aims, objectives and theoretical perspectives influenced by the completed research described here.
Start Year 2015
 
Description Conference for organisations supporting family carers in Athlone, Ireland 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact 90 minute workshop to multidisciplinary audience from all over Ireland, with guests from Bulgaria, Greece, Sweden and Poland. Around 60 delegates attended and were invited to participate in table discussions to consider the research findings and the implications for their practice. It was a lively and positive exchange
Year(s) Of Engagement Activity 2016
 
Description Two day Health Foundation seminar in Bethworth, Surrey, on measuring outcomes 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Two day seminar of 50 people including research, policy and practice representatives. Ten of the fifty were from Scotland and the findings from the research were fed directly into discussions about the direction of outcomes focused work in England.
Year(s) Of Engagement Activity 2014
 
Description Two workshops at Carers Trust Conference Scotland 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Facilitated two workshops lasting one hour each. This included presentation of workshop findings and addressed known concerns of this audience about how to record personal outcomes. Each workshop included discussion including specific question and answer exchanges with the researcher but also exchange of learning between participants
Year(s) Of Engagement Activity 2016
 
Description Workshop at national Scottish event hosted by the Health and Social Care Benchmarking Network and Social Services Research Group 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact The workshop lasted one hour and included a formal presentation and open discussion.
Year(s) Of Engagement Activity 2015
 
Description presentation (video recorded) at 6th international family carer conference in Gothenburg 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Presentation of work over many years, including the ESRC research findings
Year(s) Of Engagement Activity 2015
URL http://www.neilstewartassociates.com/sh323