HEE: Developing a reference protocol for expert elicitation in health care decision making

Lead Research Organisation: University of York
Department Name: Centre for Health Economics

Abstract

Where the budget for health care is limited, difficult decisions about the use of treatments are made by decision makers. They usually consider the benefits of each alternative treatment and select the one that is expected to maximise health. When the budget is limited, money spent on a new treatment will mean other patients may not have access to the same level of health care. A decision maker needs to weigh the health gains with the losses (cost-effectiveness). To establish cost-effectiveness, decisions are usually based on a mathematical model of the disease of interest and any treatments given. Inputs to a model, for example risk of a heart attack, are required. Due to uncertainty in the evidence, the cost-effectiveness of an intervention is often not known with certainty. However, it is important that the uncertainty in cost-effectiveness estimates is reflected in any analysis conducted. If not, any decisions can be potentially misleading.
In situations where information is unavailable, the experience of experts is essential. The process by which the beliefs of experts can be formally collected and recorded is "expert elicitation". Expert elicitation has been used in many areas including volcanology and weather forecasts. A number of alternative protocols (guides to good practice) are available for the design, conduct and use of expert elicitation, but it is not clear if any of these can appropriately be used to inform decisions using cost-effectiveness evidence.
The overall aim of this project is to establish a reference protocol for the elicitation of experts' judgements to inform health care decision making. To do this we plan a series of work packages:
WP1 Appraisal of existing protocols for eliciting distributions
We will critically review available guidance for elicitation; this will include published and unpublished literature. The principles of each method will be appraised in light of a set of requirements for health care decision making. Where the review identifies gaps in knowledge, targeted searches on the alternative methods proposed will allow advantages and disadvantages of each choice to be identified and any potential constraints to their application for cost-effectiveness analyses to be established.
WP2 Eliciting distributions to represent parameter uncertainty
In order to consider uncertainty in the health care decision making process, judgements elicited need to reflect the imperfect knowledge experts have. Within WP1 we will consider alternative methods to avoid experts expressing variability. In addition, there are alternative approaches to elicit judgements (group consensus and individual expert approaches) and these differ in terms of how well uncertainty is represented. To inform the choice of approach to elicitation we will conduct an experiment.
WP3 Understanding and appropriately characterising between-expert variation
It is important to include inputs from a range of experts in generating estimates of cost-effectiveness. These must be combined and there are alternative methods to do this: consensus and individual methods. There has been comparison of methods; however, results do not clearly support one approach over another and it is unclear which method best represents both within expert uncertainty and across expert variability. Within this work-package, we will conduct an experiment to explore the ability of consensus methods to generate expressions of uncertainty, particularly where between-expert variation is also important. The experiment will also allow us to explore different methods of pooling individual judgements.
EVALUATION: To demonstrate the usefulness of the protocol developed in WPs 1-3, we will conduct an elicitation exercise using the protocol developed to inform a decision making process in real time. York is an Assessment Group for NICEs technology and diagnostics assessment processes. We will conduct an elicitation exercise to inform a current decision.

Technical Summary

Decisions about the use of health care interventions are made in the UK. At the forefront of these decisions is the aim of maximising health, calling for judgements about the costs and effects of the interventions. Due to uncertainty in the evidence, the expected cost-effectiveness of an intervention is often uncertain. To improve the accountability of the decision making process, formal methods to quantify these judgements with uncertainty can be used (expert elicitation). The aim of this project is to establish a reference protocol for the elicitation of experts' judgements to inform health care decision making. This protocol will bring clarity over the methods for collecting and using experts' judgements within the assessment of cost-effectiveness and will demonstrate where alternative methodology may be required due to a specific context, time or costs. We plan a series of work packages, the starting point being a critical review of existing expert elicitation protocols developed outside of health care. We then propose further in-depth research guiding choices on aspects of the protocol, broadly focusing on: evaluating alternative methods of elicitation in light of the need to consider uncertainty and evaluating alternative methods of aggregation in light of the need to reflect between-expert variation due to underlying heterogeneity. In addition we will use the protocol to inform a decision making process in real time. This will achieve a number of things:
1) Establish preferred approaches for elicitation
2) Develop a guide on what quantities to elicit to inform specific parameters such as diagnostic accuracy
3) Determine which elicitation methods allow experts to express parameter uncertainty
4) Establish the accuracy of consensus-based methods in generating representations of uncertainty
5) Establish the accuracy of alternative methods of pooling the individual judgements of experts
6) Determine the applicability/usefulness of the protocol

Planned Impact

Unavoidable decisions about the use of health care interventions are made in the UK by bodies such as NICE, NHS England or Public Health England. At the forefront of these decisions, is the aim to maximise health for a particular population. With this aim in mind, judgements are required about the services or interventions that are expected to lead to higher health effects (benefits discounted of harms). The context established here is that of health care decisions made by bodies with responsibility for other patients as well as the immediate beneficiaries. It is therefore inappropriate for these bodies to abdicate responsibility for these difficult choices and be content with implicit decisions based on opaque scientific and social value judgements. Accountability and transparency are thus required. In trying to ascribe accountability, it is essential that decisions are grounded on a comprehensive evidence-base; however, decisions under uncertainty cannot avoid judgements either on matters of value - on what effects should count and how they should be measured and valued, or - on matters of fact - for example, where the evidence is contradictory on a particular aspect (input) of a cost-effectiveness assessment. In this work we do not propose exploring judgements of value, but to focus on improving the transparency of judgements relating to matters of fact. To improve the accountability of the decision making process in situation where judgements are indispensable, formal methods to quantify these beliefs with associated uncertainty can be used (expert elicitation).

Work to resolve many of the methodological uncertainties associated with generating and using expert elicitation in the context of informing cost-effectiveness is proposed here. The programme of work described is well timed given the recent early access to medicines scheme (EAMS). There is a drive towards cost reduction of the regulatory system, necessitating a need to develop procedures for expert elicitation which are focussed and transparent and make most efficient use of available expertise.

The impact of this work can only be gauged in as much as it influences practice regarding the conduct of expert elicitation in generating estimates of cost-effectiveness. To this end, our plans to hold a workshop at the end of the project and agree a protocol with key policy makers and academics, will facilitate the dissemination of the protocol to key policy makers and encourage the results of this research to be taken forward for public consultation. The workshop will include a wide range of stakeholders including members of NICE, DoH, the Gates Foundation. We will also invite contribution from the NICE citizens council and other patient/public interest groups. The final project report for the project will be available via the MRC website. We will also use our association with the Collaboration for Leadership in Applied Health Research and Care (CLAHRC) Yorkshire & Humber to additionally disseminate to a range of policy makers, academics and practitioners.

Further dissemination plans include: conference presentations, peer reviewed publications, ISPOR good practice guideline, a CostAction meeting on methodology for expert judgement in HTA, short course on expert elicitation in health care decision making and potential integration into taught and short courses led by each of the collaborating centres. The application undertaken, following development of the protocol, will also serve as an opportunity to engage with NICE. Each of these efforts will further publicise the work undertaken and maximise the opportunities for the methodologies developed to become standard practice in using expert judgements to inform health care decision making.
 
Description NICE Methods for Technology Appraisal update 
Organisation National Institute for Health and Care Excellence (NICE)
Country United Kingdom 
Sector Public 
PI Contribution Building on the evidence generated from the MRC project we reviewed the current evidence on expert elicitation and considered its use in NICE processes. We specified a set of requirements for expert elicitation conducted to inform NICE assessments. This report was used in the methods update and has been referenced.
Collaborator Contribution We worked with NICE throughout this.
Impact Report is available at: https://www.york.ac.uk/media/che/documents/Elicitation-NICE-final-report-York_01042020.pdf This is referenced in the draft NICE methods update: https://www.nice.org.uk/about/what-we-do/our-programmes/nice-guidance/chte-methods-consultation
Start Year 2020
 
Description Work with NHS Individual Funding Request Panels 
Organisation NHS England
Country United Kingdom 
Sector Public 
PI Contribution Working with NHS to explore how to express uncertainty in IFR panel meetings. Using tools developed during MRC project
Collaborator Contribution Webinars (2) and presentations to NHS England. Will be activity involved in a pilot exercise
Impact In progress Multi-disciplinary
Start Year 2019
 
Description Can we believe their beliefs: Is expert elicitation (cost) effective? Issue panel at ISPOR, Barcelona 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Laura Bojke chaired an issues panel at the International Society for Pharmacoeconomics and Outcomes Research conference, November 2018. Can we believe their beliefs: Is expert elicitation (cost) effective? Alec Morton presented the argument for the use of expert elicitation.
Year(s) Of Engagement Activity 2018
 
Description Eliciting subjective priors for cost-effectiveness modelling. Theory and Applications of Expert Judgement in Risk and Decision Analysis Workshop. University of Warwick. 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Talk by Laura Bojke on Eliciting subjective priors for cost-effectiveness modelling. Theory and Applications of Expert Judgement in Risk and Decision Analysis Workshop. University of Warwick, January 2018.
Year(s) Of Engagement Activity 2018
URL https://warwick.ac.uk/fac/sci/maths/research/events/2017-18/nonsymposium/taejrda/
 
Description Experiences of structured elicitation for cost-effectiveness analyses. Academic Unit of Health Economics, Seminar series, Faculty of Medicine and Health, University of Leeds 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Professional Practitioners
Results and Impact Talk by Marta O Soares. Experiences of structured elicitation for cost-effectiveness analyses. Academic Unit of Health Economics, Seminar series, Faculty of Medicine and Health, University of Leeds, Leeds, November 2018
Year(s) Of Engagement Activity 2018
 
Description Experiences of structured elicitation for cost-effectiveness analyses. Centre for Statistical Methodology, Seminar series,The London School of Hygiene & Tropical Medicine 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Talk by Marta O Soares. Experiences of structured elicitation for cost-effectiveness analyses. Centre for Statistical Methodology, Seminar series,The London School of Hygiene & Tropical Medicine, London, June 2018
Year(s) Of Engagement Activity 2018
 
Description Expert elicitation in Health care decision-making presentation at NICE 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Policymakers/politicians
Results and Impact Laura Bojke presented a talk on Expert elicitation in Health care decision-making at the NICE technical Forum, April 2018.
Year(s) Of Engagement Activity 2018
 
Description ISPOR Task Force 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact ISPOR Task Force on expert elicitation accepted and initiated in February 2022. Ongoing and expected to report end 2023.
Year(s) Of Engagement Activity 2022,2023
URL https://www.ispor.org/member-groups/task-forces/structured-expert-elicitation-for-healthcare-decisio...
 
Description Informing reimbursement decisions using cost-effectiveness modelling: how to generate elicited priors to capture model uncertainties. University of Nottingham seminar series, April 2017 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Professional Practitioners
Results and Impact Presentation on the topic of expert judgement for cost-effectiveness analysis to academics/students at the health economics faculty at the University of Nottingham.
Year(s) Of Engagement Activity 2017
 
Description Poster at International Society of Clinical Biostatistics 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Poster presented at International Society of Clinical Bio-statistics Annual conference, July 2019. Session titled: Structured expert elicitation (SEE) for model-based cost-
effectiveness analyses. Pesented by Linda Sharples
Year(s) Of Engagement Activity 2019
 
Description Presentation at a seminar series 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Professional Practitioners
Results and Impact Seminar delivered at the University of Manchester, Advances in Data Science Seminars. The seminar title was "How accurate are individuals in judging uncertain quantities? Findings from a novel experimental approach"
Although elicitation is widely used in crucial decision arenas, little is known about the accuracy of elicited individual estimates and the validity of the iterative feedback and revision to improve these. We here describe a novel experimental approach where the individual's knowledge is determined by observations from a simulated process allowing for accuracy to be measured. Using such an approach, three experimental set-ups were implemented. The first established that the method of elicitation (i.e. the summaries individuals are asked to provide) affects the descriptions of uncertainty provided, although the magnitude of such an effect is small in relation to the large scale of between-individual variation observed. The second found no evidence of bias or significant misrepresentation of uncertainty when individuals were asked about a quantity that required some analytic reasoning in addition to observation. The final experimental set-up showed that the likelihood of revision in Delphi-type processes depends on the method of elicitation, but it identified no evidence that less accurate individuals are more likely to revise their initial estimates or that the extent of revision is larger. The findings of this last experiment cast doubts on the benefits of the iterative process widely used within Delphi processes.
Year(s) Of Engagement Activity 2020
URL http://www.datascience.manchester.ac.uk/about/what-we-do/advances-in-data-science-seminars/previous-...
 
Description Presentation at the Fifth Subjective Bayesian Meeting 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact I presented on expert judgement in health care decision making at the Fifth Subjective Bayesian Meeting held at the Open University, Milton Keynes. This involved academics and post graduate students and comprised formal presentations followed by discussion and debate.
Year(s) Of Engagement Activity 2017
URL http://www.mathematics.open.ac.uk/seminars-and-events/annual-statistics-conference
 
Description Presentation to a special interest group on expert judgement at the University of Warwick 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact I presented a session on expert judgement in health care decision making at the MIR@W Day at the University of Warwick. This s a group of academics and post graduate students with a common interest in expert judgement.
Year(s) Of Engagement Activity 2018
URL https://warwick.ac.uk/fac/sci/maths/research/events/2017-18/nonsymposium/taejrda/
 
Description Workshop at ISPOR European meeting 2019: CONSIDERING THE APPLICABILITY OF GUIDANCE FOR EXPERT ELICITATION IN HEALTHCARE DECISION MAKING 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Workshop discussion at ISPOR European meeting. Attended by more than 5500 delegates.
Presentation of guidance and discussion of its implementation in global setting
Year(s) Of Engagement Activity 2019
 
Description Workshop at the The International Society of Pharmacoeconomics and Outcomes Research (ISPOR) Annual Meeting 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Dina Jankovic ran a workshop with the OHE and a private health economics consultancy on the use of a framework which combined value of information analysis with structured expert elicitation to aid in the reduction of decision uncertainty in health economic evaluations.
Year(s) Of Engagement Activity 2020