Ensuring test evaluation research is applicable in practice: investigating the effects of routine data on the validity of test accuracy meta-analyses

Diagnosis is a difficult process. A patient who presents to their doctor ill will often undergo a process which involves being asked questions, observed, examined and perhaps even having blood or imaging 'tests'. Each question asked or observation made is either a diagnostic test in its own right or part of one and is a necessary part of arriving at a diagnosis.

But some tests are better than others and importantly probably no test is 100% accurate. Sometimes a test result may suggest a patient has normal health when they actually have disease or have disease when they have normal health. This happens to all tests and diagnostic test accuracy research is aimed at evaluating how often this happens, in other words, determining how accurate tests are.

Essentially when a clinician decides upon a diagnosis they are consciously or otherwise invoking a probabilistic process where multiple tests are combined and the patient's diagnosis should be the one most probable given the combination of all the test results. However, for this process to be truly beneficial to the patient the clinician needs to know the accuracy of each of these tests and how likely the patient has disease before the diagnostic process has even started.

This is where the difficulty lies for those who practise evidence-based medicine. Although the accuracy of many tests has been estimated by research studies, for individual tests the accuracy may vary significantly between studies. This variation may depend on who is applying the test, how it is being applied, which patient it is being applied to and most significantly of all, how the accuracy was measured in the study. When there are several studies there are methods which allow us to combine their results. These methods may also help determine the real reasons why the test's accuracy varies. However, in general, the studies report insufficient data of sufficient quality to enable such analyses to be either possible or comprehensive.

Furthermore, from previous work, we have been able to demonstrate that in some cases the test accuracy reported by a study may be virtually impossible in some patient settings. This creates a problem for the doctor. How do they know which estimate of a test's accuracy to use if it varies greatly between studies and risks being nearly impossible for their own practice?

We have already begun to develop methods which make it possible to determine whether results from a test study are likely to accurately represent a doctor's practice in general. This would mean that a doctor could confidently apply the research to their own practice without reservation. However, sometimes the research is not reflective of the different clinical settings seen in practice and a more specific solution is required. This may be done by collecting routine data from the doctor's own setting and using it to determine a feasible range of values for the test's accuracy. This method, in its current form, is used to exclude the studies 'least likely' to derive a plausible estimate of a test's accuracy for the doctor in their own practice.

At the moment both methods are in development but potentially could be implemented into the real-world and used to improve diagnosis. There are clear patient benefits to improving diagnostic performance including reducing the number of patients treated unnecessarily and increasing the number treated appropriately. One of the aims of this research is to pilot integrating this method into General Practice to help diagnose infection. This could also help reduce the potential for antibiotic resistance by reducing the number of antibiotics prescribed inappropriately.

However, before this is done the methods need to be fully investigated to determine their utility and limitations. It may be that other approaches afford greater patient benefit, and an evaluation of these with the methods already described, will be the focus of the proposed research.

Aims
Diagnostic test accuracy (DTA) research is not always informative in practice. Primary DTA studies and meta-analyses may produce results that do not transfer to practice. Recently we developed methods that evaluate the validity of DTA meta-analyses and incorporate routine data to generate a tailored meta-analysis (TMA) estimate that is specific to the practice. Both of these need further development and evaluation.

Objectives
1. To synthesise a database for use in other work streams and review constrained models
2. To develop models which incorporate constrained data
3. To investigate methods which determine the validity of DTA review's results
4. To explore integrating the TMA model into UK General Practice

Methodology
There will be 4 overlapping work streams (WS)

WS 1
We will construct a database of different cases of DTA meta-analyses with associated routine practice data collected on the test to provide input data to WS 2-4. We will also review methods that explore modelling constrained data

WS 2
This will develop other models that include all studies but weight the less probable studies accordingly. We will investigate both a constrained maximum likelihood and a Bayesian approach. In addition, we will use covariate modelling in meta-regression analyses to explore causes of heterogeneity

WS3
The validity of meta-analysis results may be determined by combining a cross validation procedure with an appropriate method for comparing primary and secondary study estimates. We have developed one method which predicts where a new study is likely to lie, but other approaches are possible and these will be investigated

WS 4
The TMA model for test evaluations will be integrated into general practice so GPs may apply it to their patients. Templates will be designed within the electronic records system (ERS) to collect routine data. The TMA model will be developed as a web-application to access from the ERS and will be tested in 6 practices

This proposed research will produce a range of outputs that will be of interest to several different parties. In the early stages of the project, part of the communication plan will be raising the awareness of the difficulties of implementing diagnostic test accuracy (DTA) research. This will extend on our previous research into methods aimed at determining the validity and applicability of DTA studies in practice. It will also involve challenging the current orthodoxy of evidence synthesis methods in diagnostic research. As a result the outputs will inform both methodologists and clinicians.

Towards the end of the project new methods will have been developed that enable reviewers involved in the process of constructing a DTA meta-analysis to evaluate the statistical validity of the estimates produced. Moreover, the research will produce models that may synthesise estimates when the accuracy is known to be constrained in the values it may take. In this instance, the combining of evidence from the research literature with data from the practice of interest has implications for the type of statistical modelling that may be used. Consequently, the models per se will be of interest to the statisticians and methodologists in the diagnostic research community. Furthermore, constrained models are not widely known to medical statisticians so it is anticipated that the dissemination of this work is likely to yield further applications outside of diagnostic research in the future.

Whilst a large element of the research is focussed on methodological and model development the overarching theme is to enhance the decision-making on the applicability of test accuracy research in practice. Accordingly the methods will be applied to a number of diagnostic and screening tests used in clinical practice. In particular, tests used in the NHS national screening programmes such as the Nucleic acid amplification tests used to screen for Chlamydia will be evaluated and the results fed back to the respective screening committees. The outputs will be pertinent to the NHS and could potentially play a role in the future organisation of screening services.

Furthermore, one of the work streams will integrate a working model into the electronic records systems of 6 general practice surgeries so it may be used as a diagnostic decision support tool that is tailored to the patients in each practice. Overall the research will be relevant to both clinicians working at the sharp end of health care delivery and policy decision-makers who plan and implement service provision.

Patients will be the ultimate beneficiary of this research. In achieving the aim of improving decisions on when to apply test evaluation research in practice there is the potential to improve decision making on the treatment and management of patients on both the small and large scale.

The early outputs are expected to emerge in the first two years of work. Many of the outputs are likely to have an impact within 5 years of the project commencing. Thus the main model and methodological development in test accuracy research is expected to be completed within the term of the project (4 years) and disseminated within five. However, it is expected that the models will find application in other fields and although this will widen the impact it is also likely to take much longer.