Research into the methodology and the application of meta-analysis of diagnostic tests in clinical practice

Lead Research Organisation: University of Manchester
Department Name: Medical and Human Sciences


As part of the process of giving a patient a particular diagnosis, the clinician will often apply one or more diagnostic tests to the patient. On the basis of the test results he or she will judge whether the patient has a particular disease. Unfortunately diagnostic tests are rarely 100% accurate. The clinician will refer to studies which have evaluated the accuracy of the test to aid their judgment and improve the chance of making the correct diagnosis for the patient.

Often there is not just one study evaluating a test. A technique called meta-analysis has been developed to combine the results of these studies to provide an overall summary of the test‘s performance. Although used successfully as a method for combining studies on treatments of patients, meta-analysis of studies on diagnostic tests has so far provided only limited use to the practicing clinician. The aim of the research is to explore methods which may increase the utility of meta-analysis to the practicing clinician and consider alternative methods for combining studies in diagnostic research.

Technical Summary

Meta-analysis has been used successfully in therapeutics but is still developing in diagnostic research. The field has shortcomings in the effectiveness of the current techniques and in their transferability to clinical practice. The proposed research will aim to increase the utility of meta-analysis of diagnostic tests in clinical practice, and develop alternative models to combine the results of primary diagnostic studies.

1. To establish the main methodologies used in the meta-analysis of diagnostic test studies in the medical literature, identifying strengths, weaknesses and factors which influence the transfer of accurate summary estimates to clinical practice.

2. To examine all potential constraints that clinical practice may impose upon a test‘s performance characteristics and model how these may modify reported study estimates. The findings here will inform the 3rd objective.

3. To investigate and develop other potential methods and models of meta-analysis in diagnostic testing

Design and Methodology
Phase I:
A systematic review will be used as the foundation for identifying variables (e.g. co-morbidities, disease prevalence, number of test positives) which may potentially constrain the application of study results to clinical practice.

Phase II:
The utility of the identified variables will be tested. This will be done by identifying published meta-analyses (usually summary ROC curves) on specific diagnostic tests (e.g. ECG interpretation, Nucleic Acid Amplification Test for Chlamydia). Data will then be collected on the potential constraining variables relevant to a particular test. Following this the applicability of each meta-analysis to clinical practice will be assessed by combining the data collected with the summary ROC curve for that diagnostic test.

Phase III:
This final phase will be to consider other type of models for meta-analysis which allow for more than one variable (i.e. the diagnostic test thresld) to vary at any one time. This will be used as a platform to try and develop a new model which would have greater applicability in the clinical setting.

It is expected that the results of the research will help improve the understanding and utility of meta-analysis techniques on diagnostic tests particularly to the clinician who is keen to apply evidence based medicine to their practice particularly in primary care. As the work will involve developing mathematical tools and applying them to primary care, it has the added benefit of meeting two scientific priority areas for research and training as outlined on the MRC website.


10 25 50

publication icon
Willis BH (2011) The diagnostic value of symptoms for colorectal cancer in primary care. in The British journal of general practice : the journal of the Royal College of General Practitioners

publication icon
Willis BH (2011) Development and validation of a clinical prediction rule for breast cancer. in The British journal of general practice : the journal of the Royal College of General Practitioners

publication icon
Willis BH (2013) Philosophy of science and the diagnostic process. in Family practice

Description Citation in systematic reviews
Geographic Reach Multiple continents/international 
Policy Influence Type Citation in systematic reviews
Description NIHR Clinical lectureship
Amount £360,000 (GBP)
Organisation National Institute for Health Research 
Sector Public
Country United Kingdom
Start 01/2012 
End 01/2016
Title NAAT 2012 reported_Data analysis technique 
Description The research material consists of data from primary studies used in meta-analysis. Occasionally patient audit data from my own practice is used 
Type Of Material Model of mechanisms or symptoms - human 
Provided To Others? No  
Impact A paper has been accepted for publication and others should follow The work is on-going 
Title The effects of pre-test probability on the performance of clinical tests 
Description From statistical modelling and using data collected from practice, I have been able to demonstrate the effects of knowing when patients have a high probability of disease on the performance of clinical tests applied by doctors. The example used was for x-rays but it is likely that this extends to other clinical tests and has implications about the transferability of study results into practice. The research has been submitted for publication 
Type Of Material Model of mechanisms or symptoms - human 
Provided To Others? No  
Impact The research is yet to be published but it is likely that future evaluations of clinical tests and their implementation in practice will need to take into account the results of this research The work is on-going 
Title Tailored meta-analysis model 
Description The results of diagnostic studies may be wholly unrepresentative of particular practice settings. This is a method which allows us to decide which studies are representative 
Type Of Material Data analysis technique 
Provided To Others? No  
Impact None yet 
Description Collaboration on use of THIN data base to produce pharmaco-epidemiology papers 
Organisation University of Birmingham
Country United Kingdom 
Sector Academic/University 
PI Contribution Provided statistical support for the research
Collaborator Contribution Have extracted the data for analysis
Impact 3 papers
Start Year 2016
Description Collaboration with associate professor in dentistry 
Organisation Charité - University of Medicine Berlin
Country Germany 
Sector Academic/University 
PI Contribution Based on previous methodology, I was approached by colleague in Germany to apply the methods to diagnosis in dentistry. I provided the methods and data analysis
Collaborator Contribution The partner provided the data
Impact There is a potential paper but it is still under review.
Start Year 2016
Description Tailored meta-analysis 
Organisation University of Exeter
Department Medical School
Country United Kingdom 
Sector Academic/University 
PI Contribution I conceived the idea and wrote the paper.
Collaborator Contribution The collaborator advised and helped in the writing of the paper
Impact Willis BH, Hyde CJ. Estimating a test's accuracy using tailored meta-analysis - How setting-specific data may aid study selection. J Clin Epidemiol 2014; 67:538-546
Start Year 2010
Title NAAT 2012 reported_Intervention for clinical decision making 
Description The work consists of using research and live data from practice to aid diagnostic decision making. The research is still in the development stage 
Type Support Tool - For Medical Intervention
Current Stage Of Development Initial development
Year Development Stage Completed 2013
Development Status On hold
Impact Potentially there is possibility of improving diagnosis in both primary and secondary care but the methodology requires further development before this 
Description NAAT 2012 reported_Academic presentation 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? Yes
Type Of Presentation Paper Presentation
Geographic Reach Local
Primary Audience Other academic audiences (collaborators, peers etc.)
Results and Impact Around 40 academics attended who asked a number of questions and provided the opportunity for networking afterwards

The work will eventually lead to a publication
Year(s) Of Engagement Activity 2012
Description Why do diagnostic tests differ in performance between different settings? 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Type Of Presentation poster presentation
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact The presentation generated a number of questions afterwards

Discussion of a potential collaboration
Year(s) Of Engagement Activity 2013