HCD: TEST - Test Evaluation Using Structured Tools

Diagnostic tests are used to identify if a patient has a disease or health condition, which in turn is used to decide what treatment they receive. How well a test identifies a condition is described by its accuracy. Few tests are perfectly accurate, such that some patients test positive when they do not have a condition (known as a false positive results) whilst others test negative when they do have the condition (known as a false negative result). These "test errors" can lead to patients having poor outcomes, either because they do not get the treatment they need, or because they receive a treatment unnecessarily.

Guideline and policy makers such as NICE review published research studies in order to decide whether a new test should be used, looking at evidence that tests are both clinically-effective and cost-effective. Clinical-effectiveness describes how the test changes patient health, cost-effectiveness compares the change in health with the cost (or saving) involved. Most research studies that evaluate medical tests estimate the accuracy of the test. Estimates of clinical and cost-effectiveness are then obtained using mathematical modelling to predict the benefits and harms of different testing strategies according to chances of making false negative and false positive diagnoses, and the ensuing consequences of receiving appropriate or inappropriate treatment. While this approach may be fine when the tests being compared only differ in their accuracy, and treatment decisions closely follow test results, it is overly simplistic for the evaluation of many new test technologies that are being designed to improve patient health in a multitude of ways, not just by improved accuracy.

In this project we will identify the different ways in which tests can change patient health, and the different study designs and outcomes that need to be used to measure a test's impact on patient health. Many new tests report results more quickly, avoiding the need to send samples to distant laboratories and wait for the results to be reported. Other tests can be used by health professionals working in the community or primary care rather than healthcare specialists, avoiding the need to make an appointment at a hospital or clinic. These tests can benefit patients because more people can be tested, more people start treatment because it is can be prescribed straight away, and treatment is started earlier before the condition becomes more serious or (for infectious diseases) spreads to somebody else. These benefits may be large so that even when a new test is less accurate than an existing test there can be overall improvements in patient health. Also new tests may be less invasive, reducing patient discomfort and the risk of harm from testing and increasing test usage, all of which can improve patient health.

The TEST project will review existing studies that have evaluated different types of tests used for a variety of conditions to produce a classification of the ways in which tests can affect patient health. From this review we will create guidance, tools and case-studies to help researchers identify the studies that should be done and patient outcomes that should be measured to best judge how a test could change patient health, considering ways in which tests can both improve and be detrimental to health. We are producing the tools to help guideline and policy-makers to match different types of studies to different types of test evaluation question, and to help people working in the test development industry and researchers design the best studies. We will also produce tools which calculate the number of patients needed to ensure trustworthy results for different types of studies.

The project is being guided by people who are involved in producing guidelines and policy on test use, as well as by people working with the test development industry and those funding and designing new research studies

New test technologies improve patient health by changing how, when, where and in who tests are used. Current approaches to test evaluation are dominated by test accuracy studies and often ignore impacts created in other ways. We will develop tools to assist researchers and reviewers identify how a test will impact on patients, the study designs and outcomes to measure, and methods for sample size calculation.

WP1 will establish a cohort of test evaluations from reports published by HTA organisations (including NICE and WHO) and summarise the evaluation approaches used. WP2 will assess the ways new tests improve or are detrimental to patient health in a sample of test evaluations from this cohort, considering: patterns in test impact related to technology and condition; how evidence reviews consider the ways tests create impact; and where available, the magnitude of effects observed. We will select a sample representative of test technologies and health conditions, sampling 50 to 100 test evaluations until saturation of types of impact. Each test technology will be researched in depth to understand its potential to impact on patients. WP3 will use a subset of test evaluations from WP2, chosen to cover the ways tests impact on patient health. For each route we will consider alternative study designs and outcomes that measure impact, and consider trade-offs between feasibility, validity and efficiency. WP4 will identify and develop sample size calculations for each study design and outcome considered in WP3 and validate their performance in simulation studies. We will investigate and develop approaches for time-to-event outcomes (such as time to diagnosis/treatment) and for outcomes that only apply in subgroups conditioned on disease state and test result.

Project outputs will include guidance, checklists and software tools to improve the process of evidence review and generation, allowing the most clinically effective and cost effective tests to be identified.

We believe this project will have substantial impact on the quality and utility of evidence-based diagnostics, both in the UK and internationally. During our experience as test evaluation methodologists we have assisted a wide range of healthcare stakeholders to produce the best evidence for evaluating the effectiveness of diagnostic tests. We have observed a consistency in a key problem posed to us, which is an uncertainty in determining which studies need to be undertaken to provide convincing evidence of the benefits and potential harms of a new test technology. The same challenge is faced by clinicians, scientists and test developers in academia and industry, through funders of research, to national and international HTA organisations reviewing evidence and making recommendations on test provision and use. The TEST project has been conceived to address this set of methodological obstacles, both to advance and disseminate the understanding of how tests should be evaluated, and improve the quantity and quality of useful evidence for determining the effectiveness of diagnostic tests.
Direct beneficiaries to this project will therefore include:

(1) The diagnostics industry, clinicians scientists and researchers developing tests and undertaking primary research: they will be able to identify where their test fits into current care, identify the benefits and trade-offs introducing their test would require, and how their tests are likely to affect patients. This will assist them in designing their studies to focus on concepts beyond test accuracy - directly evaluating product claims as well as safety. The project outputs will provide guidance and tools to determine appropriate study designs and sample sizes.

(2) Funding bodies: who will better be able to prioritise which research to fund based on greater insight into whether it is likely to provide evidence required to change practice. The tools will help funders identify the most efficient portfolios of evidence required for test development and eliminate unnecessary or inappropriate study designs, so that they can be sure the research they fund is maximising returns in patient benefit from limited resources.

(3) National and international HTA organisations and guideline developers (such as NICE and the WHO Essential Diagnostics List): the tools will help researchers design evidence reviews to directly identify whether claims for benefits of tests are supported by appropriate evidence, whilst maintaining vigilance in considering potential unintended effects. The tools will help health economic modellers construct appropriate decision models which consider the complexity of routes by which tests may benefit or harm patients.

The ultimate impacts of the test will be on health care systems and patients. Through provision of better evidence of the clinical effectiveness of tests, health care systems will ensure that they provide effective tests and test strategies and avoid wasting resource on ineffective or even harmful tests and test strategies. Through creation of the Essential Diagnostics List (EDL) the WHO are influencing the provision of tests in member states, many of which are utilising the EDL to determine test provision. Patients who receive healthcare, who should in time benefit with improving patient health by these changes in healthcare systems around the world to focus on providing accurate, timely, necessary and useful diagnostic tests.