The REFER (REFer for EchocaRdiogram) study: a prospective validation of a clinical decision rule, NT-proBNP, or their combination, in the diagnosis of heart failure in primary care

Lead Research Organisation: University of Oxford


THE PROBLEM: Despite the benefits of early treatment, patients with heart failure (HF) are misdiagnosed by GPs who rely on diagnostic tools that have shortcomings (e.g. ECGs, chest x-rays, unstructured symptom assessment). Echocardiography is a superior test but is expensive, often delayed and limited by a lack of technicians. POTENTIAL SOLUTION: There is promising evidence that standardised questionnaires (clinical decision rules) improve clinical decision making. Using a clinical decision rule (CDR), emergency physicians improved their HF diagnosis and GPs were helped to decide if hospitalisation was needed for patients with suspected heart attacks. However, only one study has evaluated a CDR to diagnose HF in primary care, but encountered difficulties validating the rule. Another promising development is a simple blood test to prevent unnecessary referrals to echocardiography. Uncertainty about the optimal combination of symptoms and tests to include and at what level blood tests indicate the presence or absence of HF is halting progress. If a CDR is found to be accurate then further diagnostic testing for those with low CDR scores and a negative blood test is unlikely to represent a cost-effective use of resources. THE STUDY: In this diagnostic validation study we shall find out if our CDR is accurate for diagnosing HF. We have recruited 20 urban and rural GP practices to consent 500 consecutive patients aged 55 years or over presenting with possible symptoms of HF (recent onset breathlessness). After ordering a chest x-ray, GPs will refer all patients to our research clinic for a comprehensive clinical examination, including electrocardiogram, echocardiography, blood testing and recording of demographics and symptoms. Questionnaires assessing quality of life will be completed and a medical note review and follow-up quality of life questionnaires will take place at 6 and 12 months for use in a cost-effectiveness analysis of the CDR. All assessments will be carried out by qualified staff. Three cardiologists will make the final ‘gold standard’ diagnosis of HF. ETHICAL ISSUES: All assessments will be carried out within 7 days of GP referral, which is sooner than within the NHS for suspected HF, so patients will not be disadvantaged by participation. THE TEAM: The applicants are internationally respected researchers with expertise in primary care, health service research, cardiology, diagnostic testing, statistics and health economics. COSTS: Study costs are mainly to provide support for clinical assessment, study management, service user support, GP time and hiring clinic facilities.

Technical Summary

DESIGN: Prospective diagnostic validation study SETTING: 20 urban/rural general practices in Midlands Research Practices Consortium POPULATION: Adults (aged 55+), presenting to GPs with new onset symptoms of breathlessness, lethargy, or ankle oedema INVESTIGATIONS: No specific intervention will be introduced to influence GP clinical judgment. All patients undergo structured clinical assessment; data used to test performance of clinical decision rule (CDR) OUTCOME MEASURES: Primary Outcomes: 1) Observed test performance of CDR; 2) Test performance of diagnostic accuracy of NT-proBNP; 3) Proportion of LVSD and HF Secondary Outcomes: 1) Test performance of CDR and NT-proBNP; 2) Modelling of CDR test performance & most cost-effective diagnostic strategy; 3) Reliability of GP clinical judgment; 4) Reliability of clinical features; 5) Reliability of ECG interpretation; 6) Estimating best performing cut-offs for NT-proBNP; 7) Use of Echo markers of diastolic function in diagnosis of HF with preserved ejection fraction ASSESSMENT: All patients undergo structured clinical assessment, including Echo, as we have done previously in our ECHOES (Ref:2) study (see flowchart). An expert consensus panel (3 cardiologists)determine final diagnosis, based on internationally accepted criteria (Ref:1), presented in 3 steps: 1) Echo results + clinical data, except NT-proBNP and CDR variables; 2) then CDR variables provided; 3) then NT-proBNP result provided SAMPLE SIZE: 500 patients from 20 urban and rural general practices. A search of routine practice morbidity data suggest that in a practice of 6,000 patients, around 60 patients over age 55 per year will present with new onset breathlessness. Assuming a 60% response rate, it will take at least 9 months to recruit 25 such patients per practice. Calculations based on sensitivity of 94% and specificity of 48% obtained from application of the CDR in our HTA funded study (Mant et al 2009, Ref:40) and prevalence of HF in a symptomatic population of 30%. A sample size of 500 patients will be sufficient to estimate the CDR sensitivity to within 4% and specificity to within 6% at the 95% confidence level STATISTICAL ANALYSIS: CDR classification performance evaluated by comparing observed probabilities with predicted probabilities to estimate sensitivity, specificity, positive and negative predictive values (95% CIs). GP clinical judgment vs CDR; optimal NP cut-offs calculated using ROC curve analysis. Quality of life, clinical events and health care resource use will be used to estimate outcomes associated with using the CDR TIMETABLE: Ethics and R&D approvals have already been granted and the study adopted by UKCRN (No: 7944); general practices have been recruited. Months 1-6: study set up; staff and GP training. Months 7-24 patient recruitment over 18 months. Medical note review and quality of life questionnaire follow-up at 6 and 12 months. Data analysis and dissemination in final 9 months.


10 25 50

publication icon
Campbell SM (2011) Diagnostic triage and the role of natriuretic peptide testing and echocardiography for suspected heart failure: an appropriateness ratings evaluation by UK GPs. in The British journal of general practice : the journal of the Royal College of General Practitioners

publication icon
Davis RC (2012) Prevalence of atrial fibrillation in the general population and in high-risk groups: the ECHOES study. in Europace : European pacing, arrhythmias, and cardiac electrophysiology : journal of the working groups on cardiac pacing, arrhythmias, and cardiac cellular electrophysiology of the European Society of Cardiology

publication icon
Echocardiographic Normal Ranges Meta-Analysis Of The Left Heart (EchoNoRMAL) Collaboration (2014) A meta-analysis of echocardiographic measurements of the left heart for the development of normative reference ranges in a large international cohort: the EchoNoRMAL study. in European heart journal cardiovascular Imaging

publication icon
Echocardiographic Normal Ranges Meta-Analysis Of The Left Heart Collaboration (2015) Ethnic-Specific Normative Reference Values for Echocardiographic LA and LV Size, LV Mass, and Systolic Function: The EchoNoRMAL Study. in JACC. Cardiovascular imaging

Guideline Title NICE HF Management 2018 update
Description Original contributions to 2018 updated NICDE guidance on heart failure diagnosis
Geographic Reach National 
Policy Influence Type Citation in clinical guidelines
Description UK NL Collaboration 
Organisation Utrecht University
Department Institute of Earth Sciences Utrecht
Country Netherlands 
Sector Academic/University 
PI Contribution Explored a merged dataset
Collaborator Contribution Lead analyses
Impact n
Start Year 2017