HEE: Developing a reference protocol for expert elicitation in health care decision making

Where the budget for health care is limited, difficult decisions about the use of treatments are made by decision makers. They usually consider the benefits of each alternative treatment and select the one that is expected to maximise health. When the budget is limited, money spent on a new treatment will mean other patients may not have access to the same level of health care. A decision maker needs to weigh the health gains with the losses (cost-effectiveness). To establish cost-effectiveness, decisions are usually based on a mathematical model of the disease of interest and any treatments given. Inputs to a model, for example risk of a heart attack, are required. Due to uncertainty in the evidence, the cost-effectiveness of an intervention is often not known with certainty. However, it is important that the uncertainty in cost-effectiveness estimates is reflected in any analysis conducted. If not, any decisions can be potentially misleading.
In situations where information is unavailable, the experience of experts is essential. The process by which the beliefs of experts can be formally collected and recorded is "expert elicitation". Expert elicitation has been used in many areas including volcanology and weather forecasts. A number of alternative protocols (guides to good practice) are available for the design, conduct and use of expert elicitation, but it is not clear if any of these can appropriately be used to inform decisions using cost-effectiveness evidence.
The overall aim of this project is to establish a reference protocol for the elicitation of experts' judgements to inform health care decision making. To do this we plan a series of work packages:
WP1 Appraisal of existing protocols for eliciting distributions
We will critically review available guidance for elicitation; this will include published and unpublished literature. The principles of each method will be appraised in light of a set of requirements for health care decision making. Where the review identifies gaps in knowledge, targeted searches on the alternative methods proposed will allow advantages and disadvantages of each choice to be identified and any potential constraints to their application for cost-effectiveness analyses to be established.
WP2 Eliciting distributions to represent parameter uncertainty
In order to consider uncertainty in the health care decision making process, judgements elicited need to reflect the imperfect knowledge experts have. Within WP1 we will consider alternative methods to avoid experts expressing variability. In addition, there are alternative approaches to elicit judgements (group consensus and individual expert approaches) and these differ in terms of how well uncertainty is represented. To inform the choice of approach to elicitation we will conduct an experiment.
WP3 Understanding and appropriately characterising between-expert variation
It is important to include inputs from a range of experts in generating estimates of cost-effectiveness. These must be combined and there are alternative methods to do this: consensus and individual methods. There has been comparison of methods; however, results do not clearly support one approach over another and it is unclear which method best represents both within expert uncertainty and across expert variability. Within this work-package, we will conduct an experiment to explore the ability of consensus methods to generate expressions of uncertainty, particularly where between-expert variation is also important. The experiment will also allow us to explore different methods of pooling individual judgements.
EVALUATION: To demonstrate the usefulness of the protocol developed in WPs 1-3, we will conduct an elicitation exercise using the protocol developed to inform a decision making process in real time. York is an Assessment Group for NICEs technology and diagnostics assessment processes. We will conduct an elicitation exercise to inform a current decision.

Decisions about the use of health care interventions are made in the UK. At the forefront of these decisions is the aim of maximising health, calling for judgements about the costs and effects of the interventions. Due to uncertainty in the evidence, the expected cost-effectiveness of an intervention is often uncertain. To improve the accountability of the decision making process, formal methods to quantify these judgements with uncertainty can be used (expert elicitation). The aim of this project is to establish a reference protocol for the elicitation of experts' judgements to inform health care decision making. This protocol will bring clarity over the methods for collecting and using experts' judgements within the assessment of cost-effectiveness and will demonstrate where alternative methodology may be required due to a specific context, time or costs. We plan a series of work packages, the starting point being a critical review of existing expert elicitation protocols developed outside of health care. We then propose further in-depth research guiding choices on aspects of the protocol, broadly focusing on: evaluating alternative methods of elicitation in light of the need to consider uncertainty and evaluating alternative methods of aggregation in light of the need to reflect between-expert variation due to underlying heterogeneity. In addition we will use the protocol to inform a decision making process in real time. This will achieve a number of things:
1) Establish preferred approaches for elicitation
2) Develop a guide on what quantities to elicit to inform specific parameters such as diagnostic accuracy
3) Determine which elicitation methods allow experts to express parameter uncertainty
4) Establish the accuracy of consensus-based methods in generating representations of uncertainty
5) Establish the accuracy of alternative methods of pooling the individual judgements of experts
6) Determine the applicability/usefulness of the protocol

Unavoidable decisions about the use of health care interventions are made in the UK by bodies such as NICE, NHS England or Public Health England. At the forefront of these decisions, is the aim to maximise health for a particular population. With this aim in mind, judgements are required about the services or interventions that are expected to lead to higher health effects (benefits discounted of harms). The context established here is that of health care decisions made by bodies with responsibility for other patients as well as the immediate beneficiaries. It is therefore inappropriate for these bodies to abdicate responsibility for these difficult choices and be content with implicit decisions based on opaque scientific and social value judgements. Accountability and transparency are thus required. In trying to ascribe accountability, it is essential that decisions are grounded on a comprehensive evidence-base; however, decisions under uncertainty cannot avoid judgements either on matters of value - on what effects should count and how they should be measured and valued, or - on matters of fact - for example, where the evidence is contradictory on a particular aspect (input) of a cost-effectiveness assessment. In this work we do not propose exploring judgements of value, but to focus on improving the transparency of judgements relating to matters of fact. To improve the accountability of the decision making process in situation where judgements are indispensable, formal methods to quantify these beliefs with associated uncertainty can be used (expert elicitation).

Work to resolve many of the methodological uncertainties associated with generating and using expert elicitation in the context of informing cost-effectiveness is proposed here. The programme of work described is well timed given the recent early access to medicines scheme (EAMS). There is a drive towards cost reduction of the regulatory system, necessitating a need to develop procedures for expert elicitation which are focussed and transparent and make most efficient use of available expertise.

The impact of this work can only be gauged in as much as it influences practice regarding the conduct of expert elicitation in generating estimates of cost-effectiveness. To this end, our plans to hold a workshop at the end of the project and agree a protocol with key policy makers and academics, will facilitate the dissemination of the protocol to key policy makers and encourage the results of this research to be taken forward for public consultation. The workshop will include a wide range of stakeholders including members of NICE, DoH, the Gates Foundation. We will also invite contribution from the NICE citizens council and other patient/public interest groups. The final project report for the project will be available via the MRC website. We will also use our association with the Collaboration for Leadership in Applied Health Research and Care (CLAHRC) Yorkshire & Humber to additionally disseminate to a range of policy makers, academics and practitioners.

Further dissemination plans include: conference presentations, peer reviewed publications, ISPOR good practice guideline, a CostAction meeting on methodology for expert judgement in HTA, short course on expert elicitation in health care decision making and potential integration into taught and short courses led by each of the collaborating centres. The application undertaken, following development of the protocol, will also serve as an opportunity to engage with NICE. Each of these efforts will further publicise the work undertaken and maximise the opportunities for the methodologies developed to become standard practice in using expert judgements to inform health care decision making.