RobotReviewer: development and evaluation of a machine learning tool to speed up evidence synthesis in cardiovascular diseases

I am an academic GP, whose research background is in systematic reviewing. I previously worked as a Clinical Editor at the systematic reviews journal BMJ Clinical Evidence, and am about to submit my PhD thesis which looks at how best to communicate with patients about medical research about cardiovascular diseases (heart attack, stroke, and diseases of the large blood vessels).
I am applying for the Skills Development fellowship in Informatics due to a strong interest and aptitude in computer programming and statistics. This fellowship aims to further develop a computer system called RobotReviewer, which I have developed in collaboration with researchers in the US and the Netherlands.

RobotReviewer supports researchers producing a type research called a systematic review. Systematic reviews are articles which aim to summarise all relevant research on a particular topic. Systematic reviews are particularly helpful to clinicians, since it is not practical to keep on top of the vast amount of research published. Systematic reviews give a balanced view on the research, giving prominence to large and high quality studies, and examining whether research is at risk of bias.

Producing systematic reviews is laborious, taking a team 2 years on average. A large proportion of existing reviews are out of date, and reviews do not yet exist for many urgently needed topics. This is particularly the case in cardiovascular diseases, where reviews are typically out of date within 2 years. Technologies to speed up the production of systematic reviews are therefore urgently needed.

Data extraction is a key (and time-consuming) task in producing a systematic reviews, and the one on which this proposal focuses. Here, human reviewers start with all the research they intend to summarise (typically a large pile of paper documents), and identify detailed information including on how the trial was done and statistical results. These data are then entered into a standard template designed for the review. This task is done in duplicate to ensure high accuracy.

RobotReviewer works by taking in a large library of existing systematic reviews, together with articles describing clinical trials (in PDF format). RobotReviewer is able to take this data, and learn how to identify the key pieces of information in a clinical trial report which are needed to produce a systematic review. So far, RobotReviewer is able to help with the task of assessing whether clinical trials are at risk of bias. We compared the accuracy of RobotReviewer on this task against human researchers, and found that RobotReviewer was equally accurate at finding the text which discussed bias. Overall RobotReviewer was 70% accurate at judging whether a trial was biased, compared with humans who were 77% accurate.

This fellowship aims to develop the technology further, so that RobotReview is able to extract many other important pieces of information, including descriptions of the participants in clinical trials, the types of treatments they used, and data on what the benefits and harms of the treatments were.
After the technology has been developed, it will be tested, comparing its accuracy against the accuracy of human researchers doing the same task.

In order to get the technology as widely used as possible, the computer software will be released freely. Additionally, I am working with key people in the Cochrane Collaboration (an international charity who are the world's leading producer of systematic reviews), who are interested in piloting the use of the automation technology in their work.

As part of the fellowship I intend to undertake specialist training in statistics and computer science required for the project. Additionally, I will continue to collaborate with computer scientists in the US and the Netherlands on the project to develop my skills further.

Background
Systematic reviews (SRs) are the bedrock of evidence-based practice. However, due to the exponential increase in available research, it is becoming increasingly difficult to keep SRs up-to-date, and keep pace with the primary literature. This problem is compounded in the rapidly moving area of cardiovascular diseases (CVDs), where most reviews are out of date within 2 years of publication.

Aims and objectives
To extend and evaluate RobotReviewer, a machine learning (ML) system to semi-automate the extraction of data from PDFs reporting clinical trials in CVDs. The system should extract information on the population, interventions, outcomes, statistics, and risks of bias. The accuracy of the system for each variable will be evaluated via a comparison with published systematic reviews. This will allow the user to determine when the system is accurate enough for use, and to what extent.

Methodology
The system will use an emerging method, distant supervision, in which a text-mining algorithm extracts pertinent text having learned to do so from existing published systematic reviews. A corpus will be developed which links the full text of clinical trials in PDF format with structured data from the Cochrane Database of Systematic Reviews (CDSR), and ClinicalTrials.gov. Methods will be developed to label the target data elements in the PDFs using data from the CDSR and clinicaltrials.gov. ML algorithms will be developed and trained using this data. The system performance will be evaluated against accuracy standards to enable a judgment about whether the system is ready for use, and in what capacity.

Scientific and medical opportunities
For those conducting systematic reviews, an accurate automation system could reduce the huge time and cost burden in conducting SRs. For clinicians and patients, increased coverage of SRs and quicker production time would ensure that health decision-making is based on the highest quality and up-to-date evidence.

In the short-term, the major beneficiaries of the research are groups who produce evidence syntheses, who can use the tool to semi-automate systematic review production. Economically, data-extraction is a laborious task which requires highly trained (and therefore expensive) researchers to conduct. Technologies to make this process more efficient could allow UK funders to produce the same reviews for less money, or expand the scope of review topic coverage.

The Cochrane Collaboration produce close to 500 new systematic reviews per year, and with a similar number of existing reviews updated. Their current protocols rely on two researchers independently extracting data for accuracy. They have expressed enthusiasm for incorporating the technology in RobotReviewer into their work flow. We have also been approached by Dr Evidence, a US evidence synthesis firm who extract data from 7-8,000 RCTs annually. Groups such as NICE and the WHO, and other policy makers and guideline developers also extract data from large numbers of clinical trials in order to produce their guidelines. Numerous UK-based businesses such as the BMJ Group, Bazian, and Matrix expend a large amount of resources on extracting data from trial reports, and might stand to benefit from using automation technology. Use of the technology to speed up manual data-extraction in the short term is a realistic; indeed the accuracy of RobotReviewer for extracting information about risks of bias is already sufficient for this use.

In the longer term, once accuracy becomes sufficient, RobotReviewer could be used as part of a live system for clinicians and patients. If humans are no-longer needed to check the data-extractions, then they could be done instantaneously. Connecting RobotReviewer with other existing and developing technologies aimed at study identification would enable 'living' systematic reviews, and the production of evidence-based decision support systems using up-to-the-minute research.

For patients, increased availability and coverage of systematic reviews, and shortened time from protocol to publication would providing more up-to-date and comprehensive evidence to support decision making. Importantly, by use of structured data and standard ontologies, the output of the system could be used equally for multiple purposes. Such data could provide up-to-date patient information and decision aids, and ultimately help patients achieve the health outcomes which matter most to them.

Finally, the technologies this proposal aims to develop are likely to have wider applications. This proposal focuses on cardiovascular disease trials for reasons of importance, and practicality. Once applied to cardiovascular trials, the technical infrastructure would already be in place to apply to other clinical areas. The application of automation to drug trials in other areas particularly would be a comparatively small task. In the longer term, this research could provide an important step towards extracting data from other research designs.