HOD1: Inferring relative treatment effects from combined randomised and observational data

Randomised controlled trial evidence is the recognized "gold-standard" to estimate relative treatment effects to inform decision making, because patients are allocated to interventions randomly which guarantees that any patient characteristics that may be related to the patients' outcome are not related to the intervention they were allocated (since this was randomly chosen). Thus, patients randomised to different groups differ only in their allocated intervention and any difference in outcomes can be attributed to the intervention alone.

In the absence of randomisation, treatment assignment may be based on patient characteristics that also lead to a different outcome. This makes it unclear whether the different outcomes were due to the different interventions or to different patient characteristics.

Methods for combining the results of many randomised trials and obtaining a combined measure of the effect of the interventions of interest are well established. However, when randomised evidence is limited in quantity or quality, as is increasingly the case, it is natural to consider the use of non-randomised (observational) data as a substitute or supplementary source of evidence. The inclusion of observational evidence raises challenges as this evidence may not be estimating the true effect of the intervention, it may be biased.

Different methods have been proposed to handle this bias according to the origin and structure of the evidence available. However, so far, no clear consensus has emerged regarding any of the methods proposed. Thus, Health Technology Assessment bodies and particularly reimbursement agencies such as NICE face increasing challenges in their assessments of evidence and have called for extra research. To answer this call, we propose to carry out five interlinked projects: four of them are focused each on a different type of evidence structure. A fifth project develops methods that examine the impact of bias, or potential bias, on the treatment decision.

The first project will evaluate the degree of error in the treatment effects produced when including studies with a single intervention (non-comparative studies), or where the interventions of interest can be separated into two groups of interventions where no study has compared an intervention in one group with an intervention in the other group. This project relies on there being individual participant data from at least one of the studies.
The second project will consider cases where the individual participant data are not available and will (1) investigate the properties of the various methods; (2) determine which of the methods is likely to be best at reducing decision uncertainty for different kinds of observational data.
The third project will carry out computer simulations where the 'true' treatment effects can be assumed to be known and where the simulated (computer generated) data are realistically representative of the real data of interest to assess the circumstances under which the different methods are most likely to produce the least biased results.
The fourth project is specifically oriented to assess managed entry of new pharmaceuticals. There is a trend towards appraising new, promising, health technologies early, when available evidence is still limited. Methods used to inform these decisions make strong assumptions. We will explore the potential for observational data available in routinely collected databases to validate these assumptions.
The final project will extend existing methods to answer the question: "suppose this evidence is biased, how biased would it have to be before it changed our decision as to which is the best treatment?" where the decision can be based on, for example, the treatment which represents best value for money to the NHS.

Randomised controlled trial (RCT) evidence is the gold-standard to estimate relative treatment effects. Network meta-analysis (MA) is used to pool evidence from RCTs to compare multiple treatments. There is a trend towards appraising new health technologies with limited or single arm evidence, which requires adjustment methods that make strong assumptions. When RCT evidence is limited, as is increasingly the case, observational data is a potential supplementary source of evidence. We will develop methods and guidance on different ways in which observational evidence can be used in decision making in five related work packages (WP):
WP1: Development of methods to adjust effect estimates for imbalances in effect modifiers in disconnected networks of RCT, including single-arm studies. Methods to estimate the degree of error in the estimates obtained will also be developed.
WP2: We will investigate the properties of bias adjustment methods proposed in the literature in terms of their potential to estimate and adjust for bias and reduce decision uncertainty, to determine which method is likely to work best for different kinds of RCT and observational data structures.
WP3: Real world linked patient level data offers an opportunity to explore the effect of treatments in patients who may be under-represented in RCT, but are subject to selection effects and confounding. We will explore propensity scoring, Instrumental Variable and Structural Equation Modelling for the estimation of treatment effects from such data. Simulation studies will be carried out to compare the performance of these methods.
WP4: The use of registry data, adjusted for population differences and selection bias, to enhance randomised trial evidence in the context of managed entry of new pharmaceuticals will be explored.
WP5: Methods will be developed to explore how biased would the evidence (including observational evidence) have to be before it changed the decision.

Academic: Outputs from this project will be published in leading journals, presented at international conferences and local meetings, and disseminated through the network of academic colleagues the applicants regularly collaborate with. Attendance at international conferences is planned, some aimed at statistical audiences, others at health economics, decision-making and general evidence synthesis/systematic review audiences. This will ensure that the research impacts on current knowledge in the fields of Bayesian evidence synthesis, meta-analysis of randomised and observational data and causal inference, contributing to the advance of knowledge in these fields.

National Policy makers: The applicants are involved with the National Institute for Health and Care Excellence (NICE) Decision Support Unite and Guidelines Technical Support Unit which, amongst other things, provide training and support to NICE technology appraisal and guideline development groups. This involvement will continue throughout the project and the methods will be immediately available for use in NICE appraisals and guideline development, ensuring better use of the available evidence to make recommendations for clinical practice. Findings from this project will be incorporated into the training workshops currently provided to NICE, ensuring that the community of guideline developers and commissioners is aware of the potential benefits of the proposed approach, and can apply it when relevant.

International Policy makers and Industry: The applicants are involved with the delivery of 3- and 5-day courses on evidence synthesis, which are attended by researchers from the university departments contracted to undertake evaluations for NICE, by NICE analysts, and representatives of pharmaceutical companies, consultancy firms, policy makers and academic institutions from around the world. It is expected that work arising from this project will be incorporated into these courses, lectures and workshops, disseminating it to a wide audience and ensuring that it is used when relevant, impacting on policy and improving patient care worldwide. In addition, the applicants regularly advice other non-UK governments and reimbursement/HTA agencies on appropriate methods for evidence synthesis, especially in LMICs where direct RCT evidence is limited, and thus this project will have potential impact on global policy and health.

Patients and their families: Patients will benefit from methods that provide more accurate estimates of treatment benefits. The engagement with NICE, its partners in pharmaceutical and consultancy industries, will allow the methods to be applied and understood by decision-making bodies and companies submitting evidence to them, ensuring more reliable information is available to make better decision for patient care. This will result in a more efficient use of health resources, benefitting patients and society by improving health and wellbeing.