Bayesian methods, health technology assessment, and performance monitoring

Lead Research Organisation: MRC Biostatistics Unit

Abstract

Modern health care involves a large amount of data, not all of which is of high quality or is wholly appropriate for the job in hand. Nevertheless, decisions still have to made, say about the approval of new treatments or which hospitals should be inspected, and there is increasing reliance on statistical techniques for summarising the evidence in such data so that it can appropriately contribute to the decision. This programme of research is focused not only on improving these statistical techniques, but also on providing tools so that decision-makers can be assured of the robustness of the conclusions being drawn.Statistical models enable multiple data-sources to be synthesised. Our research concerns general methods for choosing between alternative models, checking whether there is conflict between data-sources, and providing guidance as to good practice. Applications include ways of allowing for differential data-quality when synthesising evidence for decisions by NICE concerning treatments to be reimbursed under the NHS, and the analysis of data on adverse drug reactions.Another area in which complex data need to be summarised for decision-making is when monitoring the quality of care within the NHS. Given the strong influence of chance variability on event rates there is a great need for robust means for identifying true signals, and yet the resulting methods must be seen to be fair and communicable to the wider community. We are exploring a mix of statistical and graphical tools for handling hundreds of indicators being regularly collected on hundreds of Trusts, and working with the Healthcare Commission on their staged implementation.

Technical Summary

The overall aim of this research programme is the provision of formal tools to help decision-making in the face of multiple sources of quantitative evidence, taking appropriate account of inadequacies and potential conflicts in available data. The underlying methodological themes explored in this programme apply whether the decision concerns, say, the approval of a new intervention by NICE, or which NHS Trust to inspect. Apart from development of the formal methods, there will be strong emphasis on dissemination to a wider audience and the provision of guidance for appropriate use. Three primary themes continue: First, research into Bayesian methods and applications will focus on improving the practice and widening the scope of Bayesian analysis in general. Model comparison is an essential component, and we shall try to refine the Deviance Information Criterion (DIC), which is becoming a standard tool in Bayesian applications. Other generic research for improving the robustness of Bayesian modelling includes a methodology for measurement of conflict in a complex graphical model, identification of prior distributions that are unintentionally informative, and tools for helping assess informative prior distributions. In addition, we shall work on extending standard modelling to include a parametric Huber distribution with a normal core and exponential tails, which appears appropriate for the analysis of micro-array data, and exploring the potential for improving robustness by cutting feedback in a full-probability model. All these innovations will be implemented in BUGS, the MCMC software that dominates applied Bayesian modelling. This is now developing as an Open Source project, and work in the Unit will focus on dissemination through articles, a tutorial book and courses. The second major theme will focus on performance monitoring in health services, motivated by the close relationship with the Healthcare Commission. Substantial statistical innovation is taking place in the Healthcare Commission in collaboration with the Unit concerning comparison, screening, surveillance and rating of organisations, and the refinement and dissemination of these methods will be essential. These use fairly basic (non-Bayesian) statistical methods, but introducing the concept of false discovery rates (FDR) appears appropriate. There will be parallel development of more sophisticated methods for monitoring multiple institutions, building on earlier work on risk-adjusted Exponentially Weighted Moving Averages (RA-EWMAs) and multiple CUSUMs. The final theme will concern the application of Bayesian ideas in healthcare evaluation, including work on adaptive studies, safety monitoring, and sensitivity analysis in complex modelling. However the primary effort will be in developing a robust approach to the quantitative assessment of possible biases in studies, both internal (lack of rigour/quality) and external (lack of relevance/generalisability). This is essentially an exercise in guided prior elicitation, but should result in a way of automatically down-weighting studies in the synthesis of evidence by bodies such as NICE. If successful, this procedure could have a major impact in the process by which evidence is used in policy decisions.

Publications

10 25 50

publication icon
Bujkiewicz S (2011) Development of a transparent interactive decision interrogator to facilitate the decision-making process in health care. in Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research

publication icon
Cooper NJ (2011) How valuable are multiple treatment comparison methods in evidence-based health-care evaluation? in Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research

publication icon
Grigg O (2007) A Simple Risk-Adjusted Exponentially Weighted Moving Average in Journal of the American Statistical Association

publication icon
Higgins JP (2009) A re-evaluation of random-effects meta-analysis. in Journal of the Royal Statistical Society. Series A, (Statistics in Society)

 
Description Analysis of health performance indicators
Geographic Reach National 
Policy Influence Type Citation in other policy documents
Impact Methodology formed the basis for the statistical technics used by the CARE Quality Commission for Surveillance of NHS Trusts.
 
Description SAGE Committee on Volcanic Ash
Geographic Reach National 
Policy Influence Type Participation in advisory committee
Impact Helped modelling on future volcanic risks from Iceland
 
Title DIC method for model comparison implemented in the BUGs software 
Description DIC method for model comparison implemented in the BUGs software 
Type Of Material Improvements to research infrastructure 
Year Produced 2006 
Provided To Others? Yes  
Impact >1000 citations to original 2002 paper in Google Scholar. Third highest cited paper in Mathematical sciences over the last 10 years in the world 
 
Title OpenBUGS diff interface 
Description Allows specification of models involving differential equations 
Type Of Material Improvements to research infrastructure 
Year Produced 2010 
Provided To Others? Yes  
Impact Ongoing research 
 
Title OpenBUGS jump interface 
Description Allows specification of variable dimension models 
Type Of Material Improvements to research infrastructure 
Year Produced 2011 
Provided To Others? Yes  
Impact Ongoing research 
 
Title OpenBUGS truncation 
Description Specification of truncated distributions 
Type Of Material Improvements to research infrastructure 
Year Produced 2010 
Provided To Others? Yes  
Impact Ongoing research 
 
Title Software for Bayesian analysis using MCMC methods 
Description Software for Bayesian analysis using MCMC methods 
Type Of Material Improvements to research infrastructure 
Year Produced 2006 
Provided To Others? Yes  
Impact Most widely used Bayesian statistical software in the world >1000 citations to User manual. Note: this tool was first provided to other researchers in 1992. 
 
Description Athens 
Organisation Agricultural University of Athens (AUA)
Department Department of Animal Science
Country Greece 
Sector Academic/University 
PI Contribution Statistical modelling and knowledge of Bayesian statistics
Collaborator Contribution Statistical modellingDomain knowledge
Impact PMID: 21937472
Start Year 2008
 
Description Athens 
Organisation University of Cambridge
Department MRC Biostatistics Unit
Country United Kingdom 
Sector Public 
PI Contribution Statistical modelling and knowledge of Bayesian statistics
Collaborator Contribution Statistical modellingDomain knowledge
Impact PMID: 21937472
Start Year 2008
 
Description BUGS 
Organisation Imperial College London
Department Department of Epidemiology and Biostatistics
Country United Kingdom 
Sector Academic/University 
PI Contribution Book writing
Collaborator Contribution Book writing
Impact Book currently under review (Nov 2011)
 
Description Care Quality Commission 
Organisation Care Quality Commission (CQC)
Country United Kingdom 
Sector Charity/Non Profit 
PI Contribution Collaboration for publications
Collaborator Contribution Collaboration for publications
Impact PMID: 19468000 Publication accepted to be published in J. Roy. Statist. Soc. Ser. A
 
Description OpenBUGS 
Organisation Merck
Country Germany 
Sector Private 
PI Contribution information exchange regarding the development of open source Bayesian software
Collaborator Contribution Collaboration on the development of open source Bayesian software
Impact information exchange regarding the development of open source Bayesian software New versions of Open Bugs software currently available as open source software
Start Year 2008
 
Description OpenBUGS 
Organisation Pfizer Ltd
Country United Kingdom 
Sector Private 
PI Contribution information exchange regarding the development of open source Bayesian software
Collaborator Contribution Collaboration on the development of open source Bayesian software Financial contribution is being used for documentation, implementation of donation mechanism and general development work
Impact Information exchange regarding the development of open source Bayesian software. New versions of Open Bugs software currently available as open source software.
Start Year 2008
 
Description Statistical research 
Organisation Novartis
Country Global 
Sector Private 
PI Contribution Collaboration on published papers on clinical trial methodology
Collaborator Contribution Collaboration on published papers on clinical trial methodology
Impact PMIDs: 19132515; PMID: 20156954; PMID: 19735071; PMID: 20156954
 
Description University of Bristol 
Organisation University of Bristol
Department School of Social and Community Medicine
Country United Kingdom 
Sector Academic/University 
PI Contribution Joint development of methodology and writing papers.
Collaborator Contribution Collaboration in new methodology for performance assessment in health care
Impact Paper accepted to be published by the J. Roy. Statist. Soc. Ser. A
Start Year 2009
 
Description University of Cambridge 
Organisation University of Cambridge
Department Department of Pure Mathematics and Mathematical Statistics
Country United Kingdom 
Sector Academic/University 
PI Contribution Joint research and writing.
Collaborator Contribution Add publications (see list marked as University!)
Impact PMID: 21903802
Start Year 2007
 
Description University of Leicester 
Organisation University of Leicester
Department Department of Epidemiology and Public Health
Country United Kingdom 
Sector Academic/University 
PI Contribution Collaboration on software and publications
Collaborator Contribution Collaboration on software and publications
Impact PMID: 21839417; PMID: 21333232; PMID: 21296599 NIHR methodology grant successfully applied for.
Start Year 2008
 
Description UoC- Health & Primary Care 
Organisation University of Cambridge
Department Department of Public Health and Primary Care
Country United Kingdom 
Sector Academic/University 
PI Contribution Methodological development and implementation
Collaborator Contribution Expert knowledgeStatistical modelling
Impact One paper ready to be submitted (2011)
Start Year 2011
 
Description UoC- Health & Primary Care 
Organisation University of Cambridge
Department MRC Biostatistics Unit
Country United Kingdom 
Sector Public 
PI Contribution Methodological development and implementation
Collaborator Contribution Expert knowledgeStatistical modelling
Impact One paper ready to be submitted (2011)
Start Year 2011
 
Description UoC- Pediatrics 
Organisation University of Cambridge
Department Department of Paediatrics
Country United Kingdom 
Sector Academic/University 
PI Contribution Methodological development and statistical modelling Supervision of MPhil student
Collaborator Contribution Data provision, expert knowledge and co-supervision of MPhil student
Impact Completed MPhil PMID: 20121899 Paper ready to be submitted (Dec 2011) One paper in preparation
Start Year 2007
 
Description UoC- Vet Med 
Organisation University of Cambridge
Department Department of Veterinary Medicine
Country United Kingdom 
Sector Academic/University 
PI Contribution Statistical modelling
Collaborator Contribution Data Plus expert knowledge
Impact Paper in preparation
Start Year 2009
 
Description UoC-Medicine 
Organisation University of Cambridge
Department Department of Medicine
Country United Kingdom 
Sector Academic/University 
PI Contribution Statistical modelling
Collaborator Contribution Data provision and expert knowledge
Impact One paper submitted One paper in preparation
Start Year 2008
 
Title OpenBUGS open source software 
Description Open source software for bayesian statistical modelling 
Type Support Tool - For Fundamental Research
Current Stage Of Development Refinement. Non-clinical
Year Development Stage Completed 2010
Development Status Under active development/distribution
Impact Widely used by statistician worldwide 
 
Description BSU WinBUGS course 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Primary Audience Public/other audiences
Results and Impact More than 10 courses done and around 350 participants have taken part

This course is part of the main BSU Short Courses
Year(s) Of Engagement Activity 2007,2008,2009,2010,2011
 
Description Bias in meta-analysis 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? Yes
Primary Audience Health professionals
Results and Impact Talk entitled "Bias in meta-analysis" presented at a workshop at NICE

Recognition for the MRC and the unit
Year(s) Of Engagement Activity 2008,2009
 
Description Big Bang Fair 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Primary Audience Public/other audiences
Results and Impact Collaborated in the preparation of exhibition presented at the Big Bang Fair in London.
More than 300 secondary school pupils visited the stand

Short video produced and is expected to be made available online together with other teaching resources
Year(s) Of Engagement Activity 2011
 
Description Cambridge Science Festival 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? Yes
Primary Audience Schools
Results and Impact Participation in the development of and help in manning the Unit display for Cambridge Science Festival. Short film produced to be available online as teaching resources for schools

Raised awareness in the local schools community for the work of the Unit
Year(s) Of Engagement Activity 2007,2008,2009,2010,2011
 
Description Introduction to Bayesian modelling using WinBUGS 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? Yes
Primary Audience Health professionals
Results and Impact 2 day course for 35-40 scientists using Bayesian modelling in WinBUGS. We have run at least 10 of these courses (since 2004) and the feedback is always overwhelmingly positive.

2-3 courses planned for next year, with a waiting list already in place.
Year(s) Of Engagement Activity 2006,2007,2008,2009,2010
 
Description Media stories 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Primary Audience Public/other audiences
Results and Impact Articles for the Times, Guardian, BBC online on health risks

300,000 hits for BBC online article
Year(s) Of Engagement Activity 2008,2009,2010,2011
 
Description Media training 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? Yes
Primary Audience Media (as a channel to the public)
Results and Impact Course for journalists on statistics

Contacts with media
Year(s) Of Engagement Activity 2008,2009,2010,2011
 
Description OpenBUGS website 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Primary Audience Public/other audiences
Results and Impact Over 30000 users at last count (2008)

NA
Year(s) Of Engagement Activity 2010,2011
 
Description Quintessential 'Counting for Health' for Parliamentarians and their researchers 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Primary Audience Policymakers/politicians
Results and Impact Policy-relevant topical examples in 'Counting for Health' were used to illustrate public understanding of risk

none
Year(s) Of Engagement Activity 2009
 
Description Radio interviews 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Primary Audience Public/other audiences
Results and Impact Radio interviews on Today programme, More or Less programme, Your and Yours programme, Analysis programme on risk and uncertainty

Increased recognition
Year(s) Of Engagement Activity 2008,2009,2010,2011
 
Description Straight Statistics 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? Yes
Primary Audience Public/other audiences
Results and Impact A campaign established by journalists and statisticians to improve the understanding and use of statistics by government, politicians, companies, advertisers and the mass media.

Better understanding of statistics
Year(s) Of Engagement Activity 2009,2010,2011
 
Description Talks to schools 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Primary Audience Schools
Results and Impact Numerous talks given to schools audiences on uncertainty and risk

No
Year(s) Of Engagement Activity 2008,2009,2010,2011
 
Description WinBUGS course 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Primary Audience Public/other audiences
Results and Impact Around 30 people attended the course

Course attendants suggested to have another course (refresher)
Year(s) Of Engagement Activity 2011