ivSyRMAF - the CAMARADES-NC3Rs in vivo systematic review and meta-analysis facility
Lead Research Organisation:
University of Edinburgh
Department Name: Centre for Clinical Brain Sciences
Abstract
Animals can be used to improve our understanding of a disease and to test the effectiveness of novel treatments. Translating findings from animal studies to humans in a clinical setting has not always been straightforward. We have pioneered the use of systematic review and meta-analysis - tools initially developed to analyse data from human clinical trials - to investigate such translational failure by analysing data from animal studies. We and others have shown that in published reports of in vivo disease modelling studies there are often flaws in their design and conduct, and there is a low prevalence of reporting of measures to reduce the risk of bias. We have also shown that studies at risk of bias tend to give inflated (higher) estimates of treatment effects.
As we have made progress, a central objective of our approach has been to support this developing field by sharing our expertise with others who wish to conduct similar reviews in their own areas of research. This involves offering methodological advice and support, a data repository, and access to our data management platform. However, we are approaching the limits of our capacity to do this, and this infrastructure funding would allow us to develop an improved database, with improved capacity for analysis and a web interface; and to provide more organised support including user manuals, online training materials and telephone and in-person advice to those wishing to conduct systematic reviews and meta-analyses.
Systematic review and meta-analysis of in vivo studies can provide evidence to change research practice; our work has identified risk of bias and publication bias in in vivo studies, led to Good Laboratory Practice guidelines for stroke modelling (2009), and helped inform the development of the ARRIVE guidelines. Our work is shared with NINDS grant awarding panels prior to funding decisions, informed the NINDS/NIH Transparency paper in Nature (2012; 16 of 64 references were to our work); and thereby informed the recent change in policy at the Nature Publishing Group, requiring reporting of risk of bias items such as randomisation, blinding, inclusion and exclusion criteria and sample size calculations.
These issues are fundamental to the 3Rs, partly because of the critical importance of getting the size of the experiment right, and secondly because an experiment at high risk of bias contributes less useful information than an experiment using the same number of animals at low risk of bias; identifying risk of bias items, and how investigators might reduce the risk of bias, is therefore critical to the 3Rs. It is our view that, while much has been achieved over the last 10 years, the utility of in vivo research could be substantially improved by the use of this technique across a much broader range of in vivo models. The purpose of this application is to allow us to be a catalyst for that much wider adoption (the demand for which is already manifest), and to allow those entering the field now to benefit from the learning and the methodological developments which we assimilated through experience.
Briefly, this will involve:(i) highlighting the importance of experimental design to improve the validity and statistical rigour of animal studies; (ii) identifying which outcome measures require fewer animals because of the variance observed, (iii) supporting the reduction of the number of animals sacrificed in studies that are too small to detect reliably the effect being sought or unnecessarily too large, , (iv) determining whether high severity tests or multiple tests are necessary, whether lengthy experiments are of added value or whether those of shorter duration are as useful, and (v) assessing publication bias where we anticipate results will be used to drive a change towards reporting of positive, negative and neutral results.
As we have made progress, a central objective of our approach has been to support this developing field by sharing our expertise with others who wish to conduct similar reviews in their own areas of research. This involves offering methodological advice and support, a data repository, and access to our data management platform. However, we are approaching the limits of our capacity to do this, and this infrastructure funding would allow us to develop an improved database, with improved capacity for analysis and a web interface; and to provide more organised support including user manuals, online training materials and telephone and in-person advice to those wishing to conduct systematic reviews and meta-analyses.
Systematic review and meta-analysis of in vivo studies can provide evidence to change research practice; our work has identified risk of bias and publication bias in in vivo studies, led to Good Laboratory Practice guidelines for stroke modelling (2009), and helped inform the development of the ARRIVE guidelines. Our work is shared with NINDS grant awarding panels prior to funding decisions, informed the NINDS/NIH Transparency paper in Nature (2012; 16 of 64 references were to our work); and thereby informed the recent change in policy at the Nature Publishing Group, requiring reporting of risk of bias items such as randomisation, blinding, inclusion and exclusion criteria and sample size calculations.
These issues are fundamental to the 3Rs, partly because of the critical importance of getting the size of the experiment right, and secondly because an experiment at high risk of bias contributes less useful information than an experiment using the same number of animals at low risk of bias; identifying risk of bias items, and how investigators might reduce the risk of bias, is therefore critical to the 3Rs. It is our view that, while much has been achieved over the last 10 years, the utility of in vivo research could be substantially improved by the use of this technique across a much broader range of in vivo models. The purpose of this application is to allow us to be a catalyst for that much wider adoption (the demand for which is already manifest), and to allow those entering the field now to benefit from the learning and the methodological developments which we assimilated through experience.
Briefly, this will involve:(i) highlighting the importance of experimental design to improve the validity and statistical rigour of animal studies; (ii) identifying which outcome measures require fewer animals because of the variance observed, (iii) supporting the reduction of the number of animals sacrificed in studies that are too small to detect reliably the effect being sought or unnecessarily too large, , (iv) determining whether high severity tests or multiple tests are necessary, whether lengthy experiments are of added value or whether those of shorter duration are as useful, and (v) assessing publication bias where we anticipate results will be used to drive a change towards reporting of positive, negative and neutral results.
Technical Summary
Systematic review and meta-analysis can provide empirical evidence of the impact of internal and external validity of in vivo studies across research domains. We spend substantial time supporting those who wish to conduct such reviews in their own fields through methodological advice and support, a data repository, and access to our database.
This database can now be used in the analysis of diverse data, from animal models of stroke to receptor binding assays. It includes data from over 3,000 publications involving over 80,000 animals. Currently, we curate data on MS Access, accessed by external users via a secure remote desktop connection; migration to an SQL server based database will improve system performance, scalability and accessibility. Providing web based access backed up by the support of a research will substantially improve our effectiveness in supporting such activity. We will also provide user manuals, online training materials and telephone advice.
Our support will include:
i) Systematic reviews and meta-analyses of preclinical studies, with calculation of summary effect sizes and determination in that model system of the impact of study design features and risk of bias items.
ii) For a given disease model, identification of the outcome measure with the smallest variance. This will help reduce the number of animals used in future studies.
iii) Providing information for power calculations, so that the number of animals sacrificed in studies are neither too small to detect reliably the effect being sought or unnecessarily large.
iv) Determine whether high severity, multiple and lengthy experiments provide more information than low severity, single, short experiments. This will inform ethical considerations about the choice of measures to be used, supporting a reduction in the suffering of the animals.
v) Secondary analyses investigating issues such as publication bias, and improvements in the quality of study reports over time.
This database can now be used in the analysis of diverse data, from animal models of stroke to receptor binding assays. It includes data from over 3,000 publications involving over 80,000 animals. Currently, we curate data on MS Access, accessed by external users via a secure remote desktop connection; migration to an SQL server based database will improve system performance, scalability and accessibility. Providing web based access backed up by the support of a research will substantially improve our effectiveness in supporting such activity. We will also provide user manuals, online training materials and telephone advice.
Our support will include:
i) Systematic reviews and meta-analyses of preclinical studies, with calculation of summary effect sizes and determination in that model system of the impact of study design features and risk of bias items.
ii) For a given disease model, identification of the outcome measure with the smallest variance. This will help reduce the number of animals used in future studies.
iii) Providing information for power calculations, so that the number of animals sacrificed in studies are neither too small to detect reliably the effect being sought or unnecessarily large.
iv) Determine whether high severity, multiple and lengthy experiments provide more information than low severity, single, short experiments. This will inform ethical considerations about the choice of measures to be used, supporting a reduction in the suffering of the animals.
v) Secondary analyses investigating issues such as publication bias, and improvements in the quality of study reports over time.
Planned Impact
ivSyRMAF will have a major impact on reduction and refinement, and may have a lesser impact on replacement.
On Reduction, Russell and Burch were concerned that animal experiments should be hypothesis testing, rather than exercises in screening the efficacy of large chemical libraries. Few such in vivo screening experiments now occur, and the principle of Reduction is perhaps better understood as using the minimum number of animals required reliably to test a hypothesis, to get the maximum reliable information from each animal. Very few (less than 1 in 50) publications report a sample size calculation; by supporting systematic reviews of in vivo experiments ivSyRMAF will provide reliable data to support sample size calculations for various outcome measures. For the latter, systematic reviews and meta-analyses of a field of research will identify study design features which compromise potential clinical application or which increase the risk of bias, and will also allow the detection of publication bias. We expect that this will lead to a series of empirically based good laboratory practice guidelines specific to different disease models.
On Refinement, the ivSyRMAF will allow comparison of the statistical performance of different outcome measures. All other things being equal, investigators should choose the outcome measure with the greatest precision.
Different outcome measures do measure different aspects of biology, but since many experiment report more then one outcome, and because a diverse range of biological activity is usually represented in the candidate interventions, we can use regression analysis to understand the relationship between different outcomes at different times. For instance, we have shown that in models of multiple sclerosis histological evidence of demyelination is a good surrogate for neurobehavioural outcome in the first week, but after the second week this relationship is lost, and it is axon loss which best predicts neurobehavioural outcome.
We can therefore start to characterise the extent to which subjecting the animal to an additional test results in additional knowledge. As a further example, in a systematic review of spinal cord injury we found that the most commonly used neurobehavioural test (an open-field locomotion test) gave the same information as multiple tests, and the results of these single tests were more precise.
Importantly, it will also be possible to ascertain whether a test of lower pain, suffering, distress or lasting harm provides the same information as other tests, and whether there is a trade-off in increased sample size. One benefit of this approach is that it uses existing data, rather than requiring additional animal experiments to develop and validate the approach.
On Replacement, it may be that in some research fields the animal models used are so imprecise, so at risk of bias, and can be shown to be such poor predictors of the human disease being modelled that in vivo models should not, until these issues have been addressed, be used; and that they should be replaced with other approaches.
Sena leads and Macleod is a member of the Multi-Part consortium, recently awarded FP7 funding to establish a framework for multicentre studies in focal ischaemia, with a clear expectation that the framework should also be relevant for other in vivo disease models. The standard operating procedures (SOPs) which Multi-Part will develop will include attention to sample size calculations and the choice of outcome measure, and so there are clear synergies with the work described here. Where other in vivo disease models might also benefit from a multicentre approach we will encourage those conducting reviews in those areas to engage with Multi-part, that their work might inform the development of SOPs for multicentre studies in their disease area.
On Reduction, Russell and Burch were concerned that animal experiments should be hypothesis testing, rather than exercises in screening the efficacy of large chemical libraries. Few such in vivo screening experiments now occur, and the principle of Reduction is perhaps better understood as using the minimum number of animals required reliably to test a hypothesis, to get the maximum reliable information from each animal. Very few (less than 1 in 50) publications report a sample size calculation; by supporting systematic reviews of in vivo experiments ivSyRMAF will provide reliable data to support sample size calculations for various outcome measures. For the latter, systematic reviews and meta-analyses of a field of research will identify study design features which compromise potential clinical application or which increase the risk of bias, and will also allow the detection of publication bias. We expect that this will lead to a series of empirically based good laboratory practice guidelines specific to different disease models.
On Refinement, the ivSyRMAF will allow comparison of the statistical performance of different outcome measures. All other things being equal, investigators should choose the outcome measure with the greatest precision.
Different outcome measures do measure different aspects of biology, but since many experiment report more then one outcome, and because a diverse range of biological activity is usually represented in the candidate interventions, we can use regression analysis to understand the relationship between different outcomes at different times. For instance, we have shown that in models of multiple sclerosis histological evidence of demyelination is a good surrogate for neurobehavioural outcome in the first week, but after the second week this relationship is lost, and it is axon loss which best predicts neurobehavioural outcome.
We can therefore start to characterise the extent to which subjecting the animal to an additional test results in additional knowledge. As a further example, in a systematic review of spinal cord injury we found that the most commonly used neurobehavioural test (an open-field locomotion test) gave the same information as multiple tests, and the results of these single tests were more precise.
Importantly, it will also be possible to ascertain whether a test of lower pain, suffering, distress or lasting harm provides the same information as other tests, and whether there is a trade-off in increased sample size. One benefit of this approach is that it uses existing data, rather than requiring additional animal experiments to develop and validate the approach.
On Replacement, it may be that in some research fields the animal models used are so imprecise, so at risk of bias, and can be shown to be such poor predictors of the human disease being modelled that in vivo models should not, until these issues have been addressed, be used; and that they should be replaced with other approaches.
Sena leads and Macleod is a member of the Multi-Part consortium, recently awarded FP7 funding to establish a framework for multicentre studies in focal ischaemia, with a clear expectation that the framework should also be relevant for other in vivo disease models. The standard operating procedures (SOPs) which Multi-Part will develop will include attention to sample size calculations and the choice of outcome measure, and so there are clear synergies with the work described here. Where other in vivo disease models might also benefit from a multicentre approach we will encourage those conducting reviews in those areas to engage with Multi-part, that their work might inform the development of SOPs for multicentre studies in their disease area.
