Identifying relevant studies for systematic reviews and health technology assessments using text mining

Systematic reviews are a widely used method to bring together the findings from multiple studies in a reliable way, and are often used to inform policy and practice (such as guideline development). A critical feature of a systematic review is the application of scientific method to uncover and minimise bias and error in the selection and treatment of studies. However, the large and growing number of published studies, and their increasing rate of publication, makes the task of identifying relevant studies in an unbiased way both complex and time consuming.

Unfortunately, the specificity of sensitive electronic searches of bibliographic databases is low. Reviewers often need to look manually through many thousands of irrelevant titles and abstracts in order to identify the much smaller number of relevant ones; a process known as 'screening'. Given that an experienced reviewer can take between 30 seconds and several minutes to evaluate a citation, the work involved in screening 10,000 citations is considerable (and the burden of screening is sometimes considerably higher than this).

The obvious way to save time in reviews is simply to screen fewer studies. Currently, this is usually accomplished by reducing the number of citations retrieved through electronic searches by developing more specific search strategies, thereby reducing the number of irrelevant citations found. However, limiting the sensitivity of a search may undermine one of the most important principles of a systematic review: that its results are based on an unbiased set of studies.

We therefore propose to develop and evaluate an alternative approach which addresses both of these issues: it is important to have as sensitive a search as is possible, as this is necessary to obtain reliable review findings; but it is also sometimes impossible to screen the number of citations that these sensitive searches will generate. Thus, some form of automation is needed to identify the citations that do, and do not, need to be screened manually. As the data upon which the automation must work are in the form of text, we are looking to the relatively new science of text mining to provide solutions to these problems.

There are two ways of using text mining that are particularly promising for assisting with screening in systematic reviews: one aims to prioritise the list of items for manual screening so that the studies at the top of the list are those that are most likely to be relevant ('screening prioritisation'); the second method uses the manually assigned include/exclude categories of studies in order to 'learn' to apply such categorisations automatically ('automatic classification').

We know of no existing evaluations of screening prioritisation. There are a small number of other groups developing tools for automatic classification, but this project adds value by: implementing the technology in ongoing reviews; developing metrics for their use such reviews; and engaging with systematic reviewers and computer scientists with a view to building capacity for further implementation and development.

As the use of these technologies and the development of validated methods for their use are in their infancy, an important part of the project is outreach: to build interest, capacity and enthusiasm for their use in the future.

By reducing the burden of screening in reviews, new methodologies using text mining may enable systematic reviews to both: be completed more quickly (thus meeting exacting policy and practice timescales and increasing their cost efficiency); AND minimise the impact of publication bias and reduce the chances that relevant research will be missed (by enabling them to increase the sensitivity of their searches). In turn, by facilitating more timely and reliable reviews, this methodology has the potential to improve decision-making across the health sector and beyond.

There are two main components to this study: 1) a retrospective analysis of data from existing reviews; and 2) a prospective analysis involving the use of text mining in ongoing reviews. These analyses will be used to evaluate two new methods: screening prioritisation and automatic classification. As the use of these technologies and the development of validated methods for their use are in their infancy, an important part of the project is outreach: to build interest, capacity and enthusiasm for their use in the future.

The retrospective analyses will simulate the conduct of screening using text mining utilising data from six completed EPPI-Centre reviews and subsequently from between five and eight ongoing reviews. Learning from the retrospective simulation studies will inform the parameters selected for the prospective studies and also to develop tools and metrics to evaluate their performance.

Both text mining techniques will be available for prospective evaluation in EPPI-Centre reviews over the period of this project and by arrangement with reviews conducted by external organisations too. In selected reviews, searches will be more extensive than usual and we will also maintain a record of the studies that would have been identified using standard search techniques for comparison.

In a systematic review we are less interested in predictive performance (the standard way of evaluating a classifier), but in the ability of the system - including human interaction - to identify all relevant studies as efficiently as possible. Wallace and colleagues have suggested two additional parameters, which we propose to use in addition to standard metrics: yield (the proportion of relevant studies identified) and burden (the total number manually screened). In reviews that screen everything manually, yield and burden are 100%. Successful automated approaches will reduce the burden of manual screening whilst retaining a yield of 100%.

SOCIAL AND ECONOMIC IMPACT
By reducing the burden of screening in reviews, new methodologies using text mining may enable systematic reviews to both: be completed more quickly (thus meeting exacting policy and practice timescales); AND minimise the impact of publication bias and reduce the chances that relevant research will be missed (by enabling them to increase the sensitivity of their searches). In turn, by facilitating more timely and reliable reviews, this methodology has the potential to improve decision-making across the health sector and beyond.

Thus, while the immediately apparent direct impact will be evident for researchers, as outlined in 'Academic beneficiaries', the dual benefits of increased methodological rigour and increased time- and cost-efficiency will also have direct effects for policymakers, those that commission systematic reviews and those that affected by their decisions.

The proposed research is likely to generate commercially exploitable results through the integration of the text mining tools in the software applications and the supply of support in using these technologies.

ENSURING IMPACT THROUGH DISSEMINATION
Impact will be achieved through the 'Communications plan' and the strategy outlined in detail in the 'Pathways to impact' document. The project goes beyond mere 'dissemination' and aims to engage key potential user communities in its research and evaluation process.