Evidence synthesis of diagnostic test performance from a decision-making perspective

A diagnostic test is any kind of medical test or assessment used to determine whether a patient does or does not have a disease. A highly accurate, but also highly invasive and/or expensive, test for a disease might exist (the 'gold standard' or reference test). The accuracy of less invasive and/or expensive tests ('index tests') is obviously of great interest. Assuming that the reference test has correctly classified all patients, the accuracy of the index test can be quantified using two measures: (1) the sensitivity, defined as the proportion of diseased individuals who correctly test positive on the index test, and (2) the specificity, the proportion of non-diseased individuals who correctly test negative.

In practice, most index tests do not directly deliver a 'disease' versus 'no disease' outcome. Often the test delivers a number on a continuous scale, for example the concentration of some substance in the blood. A patient is classified as diseased if his/her test result is greater than some cut-off value. As this value is reduced over its possible range, sensitivity is increased, but at the cost of reduced specificity. A graph of this relationship is called the Receiver Operating Characteristic (ROC) curve. This can often be drawn even for tests without an explicit continuous measure since, for example, some clinicians interpreting images will tend to have a stricter definition of 'disease' than others.

Often estimates of the sensitivity and specificity of a test are available from multiple studies. As in other areas of medicine (e.g. treatment effectiveness), it is desirable to pool these estimates, to summarise all available data. This is called 'meta-analysis'. Meta-analysis of diagnostic test accuracy is often considered more complex than that in other areas, due to there being two dimensions of test accuracy. Further, although variability in e.g. patient populations and study designs is a concern throughout all areas of meta-analysis, for diagnostic test accuracy there is also the specific concern that studies are likely to have used different cut-off values. These meta-analyses therefore tend to exhibit a high degree of between-study variability. This leads to inconclusive summary evidence on test accuracy, and difficulty in making decisions or recommendations about the best testing strategies.

Standard methods for meta-analysis of diagnostic test accuracy pool only a single pair of sensitivity and specificity from each study. However, often studies report multiple points on the ROC curve. In addition, studies often report the cut-off value(s) used, but these are not usually incorporated into the model. Intuitively, these two types of information, both usually discarded, offer great potential to explain some of the between-study variability, helping us to make better sense of the evidence and, in particular, to choose the best cut-off.
Meta-analysis methods focus on producing pooled estimates of sensitivity and specificity, or a 'summary' ROC curve. But, in practice, important decisions such as whether to test, which test to use and at which cut-off should also be based on other information: e.g. the effectiveness of treatments to be provided in light of test results, the amount of disease in the population, and relevant costs. The cost-effectiveness of testing strategies can be quantified by a decision model, but there are questions regarding how best to inform this model from the meta-analysis results. In addition, there has been relatively little work to date on methods for choosing between alternative tests.

In this project, I will evaluate some sophisticated methods that have been suggested in this research area. In addition, I will work on further methods development for: (i) making full use of 'all available evidence' on testing (ii) how best to inform a decision model based on a meta-analysis of test accuracy, (iii) modelling test comparisons and choosing between multiple diagnostic tests.

Aims: To evaluate, improve and develop methods for (i) making better use of 'all available data' in meta-analyses of diagnostic test accuracy, (ii) informing an economic decision model based on the synthesised evidence and (iii) comparing and choosing between competing diagnostic tests.

Objectives:
1) To investigate the circumstances in which inferences are improved by accounting for between-study correlation in sensitivity and specificity.
2) To evaluate, and potentially to extend, methods for incorporating multiple points on the ROC curve from some studies into a meta-analysis.
3) To develop methods for incorporating explicit threshold values into the meta-analysis model, when available.
4) To investigate a range of methods for informing the decision model, and produce guidelines on the appropriate approach in different scenarios.
5) To develop methods for network meta-analysis of diagnostic test accuracy.
6) To investigate the validity of test comparisons based on less-than-ideal data, for example when tests A and B have only been evaluated in separate sets of studies.
7) To perform state-of-the-art analyses of multiple real examples.
8) To develop a 2-3 day short course, to disseminate high quality methodology.

Methodology: I envision completing several of these objectives via extensions of the Rutter and Gatsonis (Statistics in Medicine, 2001) model for meta-analysis of diagnostic test accuracy, which is one of several suggested parameterisations for bivariate meta-analysis in this area. I will also draw on the literature on signal detection theory, economic decision analyses, and network meta-analysis.

Scientific Opportunities:
1) The availability of improved methodology in this area to statistical analysts
2) Better summary evidence on the accuracy of diagnostic tests, and the relative accuracy of competing tests.
3) Improved, more cost-effective, decision-making regarding testing strategies
4) My own professional development

As described in the Case for Support, the current standard methods for evaluation of diagnostic tests are sub-optimal. In particular, many evidence syntheses in this area 'throw away' a lot of potentially informative data. The very high levels of unexplained heterogeneity that are often observed in meta-analyses of diagnostic test accuracy are likely to be due in part to this. Strengthened methods for analysing test performance, comparing tests and deciding how to use tests will lead to better decision making. Ultimately, patients and clinicians will benefit from improved diagnosis, and the NHS from more cost-effective use of health resources.

More directly, methods development in this area will benefit all those who have a role in making recommendations about the use of diagnostic tests: In the UK this is the National Screening Committee, the NICE Diagnostic Advisory Committee, and the NICE Centre for Clinical Practice, who are responsible for NICE Clinical Guidelines (CGs), along with the National Collaborating Centres who carry out most of the work behind CGs. In addition, manufacturers of diagnostic devices would be able to use improve methodologies in their submissions to bodies such as NICE. NICE is considered to be one of the leading organisations of its type world-wide both in terms of its processes and the excellence of its methodology. Statistical methods used at NICE have the potential to have a far-ranging impact.

In the USA, recommendations are made largely through the Agency for Healthcare Research and Quality (AHRQ). I have planned two research and training visits to the Centre for Evidence Based Medicine at Brown University, which hosts a centre designated by the AHRQ to perform technology appraisals, with a particular focus on diagnostic tests. These visits, and subsequent collaborations, will facilitate my research having an impact on AHRQ guidance for analysis of tests.

My research will also benefit the Cochrane Collaboration, and other bodies performing systematic reviews and meta-analyses of diagnostic test performance data.

I will benefit from improved research skills, a greatly increased knowledge of health economics and the processes involved in making healthcare decisions and recommendations, and the development of multiple collaborations. Each of these factors will help me to establish myself as an independent researcher, with the potential for leading subsequent programs of research and managing teams. For example, during my fellowship I aim to learn about the complex issues regarding evaluating tests and screening in the absence of a 'gold standard' test. There is a clear need for improved methodology in this area, and I envision this initial research leading to a grant or further fellowship application in the future.