Evaluation and parameterisation of individual-based models of animal populations
Lead Research Organisation:
University of Reading
Department Name: Sch of Biological Sciences
Abstract
Ecosystems are populated by autonomous, adaptive individuals, each figuring out its own ways of achieving its goals. It is a widely shared hope that the general principles governing such complex systems will eventually be understood from analysis of computer simulations known collectively as individual-based models (IBMs). IBMs are dynamical systems containing many autonomous interacting agents which are used where, broadly, the factors influencing the behaviour of individual agents are known, but interest centres on what happens at the population level. Will the population increase or decrease? How fast will be the response? Where practical management of ecosystems is required, many consider this can only be realistically performed with IBMs. Examples include conservation management of nature reserves and shell fisheries, assessment of environmental impacts of building proposals including wind farms and highways, management of fish stocks and assessment of the effects on non-target organisms of new chemicals for the control of agricultural pests. Articles in scientific journals have suggested IBMs are the only realistic way forward in diverse fields including economic analysis where the recent global 'credit crunch' might have been avoided with the use of such models. Thus IBMs are the only practicable method of modelling many complex systems where prediction is of vital importance to all.
Despite the widely-appreciated importance of IBMs, the evaluation of these very complex systems still leaves much to be desired. Clearly, the purpose of a model is to explain the world that we see around us. From a statistical point of view we wish to 'fit' the model to data. How can we do this? Recent advances in statistical theory, known as Approximate Bayesian Computation, ABC, suggest how this might be done. Implementation of ABC requires development of practical methods that will allow users to fit their IBM models to real data in an efficient manner. This Bayesian approach should allow calculation of distributions of possible parameter values in IBMs, given observations, and evaluation of whether one model is better than another. In this project we devise practical methods that will allow all makers of IBMs to validate their models properly by reference to relevant data. Provision of such methods is crucial if we are to have robust and reliable bases for making crucial decisions about environmental impacts, nature conservation, and the licensing of new chemicals for the control of agricultural pests.
Despite the widely-appreciated importance of IBMs, the evaluation of these very complex systems still leaves much to be desired. Clearly, the purpose of a model is to explain the world that we see around us. From a statistical point of view we wish to 'fit' the model to data. How can we do this? Recent advances in statistical theory, known as Approximate Bayesian Computation, ABC, suggest how this might be done. Implementation of ABC requires development of practical methods that will allow users to fit their IBM models to real data in an efficient manner. This Bayesian approach should allow calculation of distributions of possible parameter values in IBMs, given observations, and evaluation of whether one model is better than another. In this project we devise practical methods that will allow all makers of IBMs to validate their models properly by reference to relevant data. Provision of such methods is crucial if we are to have robust and reliable bases for making crucial decisions about environmental impacts, nature conservation, and the licensing of new chemicals for the control of agricultural pests.
Planned Impact
All those making individual-based models (IBMs) and using them for planning purposes will benefit. This includes planners and managers who use IBMs for conservation management of nature reserves and shell fisheries, assessment of environmental impacts of building proposals including wind farms and highways, management of fish stocks and assessment of the effects on non-target organisms of new chemicals for the control of agricultural pests. Examples of organisations who use IBMs are the UK Chemicals Regulation Directorate, Syngenta, Bayer, Dow Agrochemicals, and the Environment Agency. Other areas may eventually benefit - over the past 30 years ecology has produced over a thousand IBMs. Individual-based modelling is the only modeling method that can be used where explicit spatial landscapes affect population dynamics. As suggested in recent commentaries in Nature there is a view that current economic modelling methods are unsuitable and that individual-based modelling is the way forward.
All IBM modellers need to validate their models to show that they are reliable. The principal obstacle to validation is lack of a sound statistical methodology. Currently modellers rely on Pattern-Oriented Modelling but this has many drawbacks (lack of standard errors, ambiguity about best model) and modellers would prefer to have methods of evaluation like those used in statistics, where statistical tests are available to guide choice between models, statistics such as R2 or deviance can be used to assess overall goodness of fit, and parameters can be calibrated from data and confidence intervals reported using least-squares or maximum-likelihood procedures.
The proposal has been designed to meet these needs. Our research programme addresses two key areas of concern in IBMs: parameterisation and model validation. By providing a method of solving these problems, ecology as a whole will benefit through more accurate modelling of ecosystems with potential for improved forecasting, management and environmental risk assessment. In the longer term the methods developed may also allow more accurate prediction of the future performance of the economy and geographyical systems. We will create and maintain a website detailing algorithms, scripts, programs and manuals to allow widespread implementation of our methodology by IBM modellers.
We hope the benefits described here will be fully realised within four years. Our research will contribute by implementing and demonstrating practical methods for parameterising and validating IBMs. This will facilitate the production of credible and trustworthy models for use throughout environmental planning. Environmental planning contributes to the nation's health by regulating the chemicals used in agriculture and to the nature's culture and quality of life through improved conservation and management of nature reserves and sites of special scientific interest.
All IBM modellers need to validate their models to show that they are reliable. The principal obstacle to validation is lack of a sound statistical methodology. Currently modellers rely on Pattern-Oriented Modelling but this has many drawbacks (lack of standard errors, ambiguity about best model) and modellers would prefer to have methods of evaluation like those used in statistics, where statistical tests are available to guide choice between models, statistics such as R2 or deviance can be used to assess overall goodness of fit, and parameters can be calibrated from data and confidence intervals reported using least-squares or maximum-likelihood procedures.
The proposal has been designed to meet these needs. Our research programme addresses two key areas of concern in IBMs: parameterisation and model validation. By providing a method of solving these problems, ecology as a whole will benefit through more accurate modelling of ecosystems with potential for improved forecasting, management and environmental risk assessment. In the longer term the methods developed may also allow more accurate prediction of the future performance of the economy and geographyical systems. We will create and maintain a website detailing algorithms, scripts, programs and manuals to allow widespread implementation of our methodology by IBM modellers.
We hope the benefits described here will be fully realised within four years. Our research will contribute by implementing and demonstrating practical methods for parameterising and validating IBMs. This will facilitate the production of credible and trustworthy models for use throughout environmental planning. Environmental planning contributes to the nation's health by regulating the chemicals used in agriculture and to the nature's culture and quality of life through improved conservation and management of nature reserves and sites of special scientific interest.
Publications
Van Der Vaart E
(2015)
Calibration and evaluation of individual-based models using Approximate Bayesian Computation
in Ecological Modelling
Watson J
(2022)
Incorporating environmental variability in a spatially-explicit individual-based model of European sea bass?
in Ecological Modelling
Van Der Vaart E
(2016)
Predicting how many animals will be where: How to build, calibrate and evaluate individual-based models
in Ecological Modelling
Van Der Vaart E
(2018)
Taking error into account when fitting models using Approximate Bayesian Computation.
in Ecological applications : a publication of the Ecological Society of America
Description | We have shown that a simple form of the statistical technique we are working on, ABC, can, in some circumstances, parameterise and evaluate complex computer models. Further work has discovered improvements that improve the calibration and evaluation. We have shown that the computed parameter values and credible intervals are accurate according to the best available diagnostic test. |
Exploitation Route | After further exploration and development of alternative techniques, ABC should be useful to all those using complex computer models, provided there is data available with which to evaluate how well model outputs match data. |
Sectors | Aerospace Defence and Marine Agriculture Food and Drink Environment Financial Services and Management Consultancy Government Democracy and Justice Manufacturing including Industrial Biotechology Transport |
URL | https://ibmreading.wordpress.com/about/ |
Description | Undertaking ABC on IBM for CEFAS (fish community dynamics model) |
Organisation | Centre For Environment, Fisheries And Aquaculture Science |
Country | United Kingdom |
Sector | Public |
PI Contribution | Method of evaluating computer models |
Collaborator Contribution | Example of application of our methods |
Impact | None yet |
Start Year | 2014 |
Description | Undertaking ABC on IBM for CEH (midge population dynamics |
Organisation | UK Centre for Ecology & Hydrology |
Country | United Kingdom |
Sector | Public |
PI Contribution | Methods for evaluation of computer model |
Collaborator Contribution | New example of application of our methods |
Impact | No impact yet |
Start Year | 2014 |
Description | Undertaking ABC on simple mechanistic model for Syngenta (pesticide risk assessment) |
Organisation | Syngenta International AG |
Department | Syngenta Ltd (Bracknell) |
Country | United Kingdom |
Sector | Private |
PI Contribution | Methods for evaluating computer models |
Collaborator Contribution | Example of use of our methods |
Impact | None yet |
Start Year | 2014 |
Description | "ABC for IBM" Workshop (11th July 2014), Charles Darwin House, London |
Form Of Engagement Activity | A formal working group, expert panel or dialogue |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Sparked interest and discussion Induced others to try the methods we developed. |
Year(s) Of Engagement Activity | 2014 |
Description | Created a website (9th July 2014; www.abc4ibm.com) to disseminate information on the method |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Provided information to practioners none so far |
Year(s) Of Engagement Activity | 2014 |
URL | http://www.reading.ac.uk/biologicalsciences/res/eeb/ecology/biosci-ibm.aspx |
Description | One-day workshop for the British Ecological Society Annual Meeting |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Postgraduate students |
Results and Impact | Demonstrated our methods of building, calibrating and evaluating Individual Based Models to 60 attendees of the British Ecological Society Annual Meeting in an all-day workshop. Much interest including subsequent email enquiries. |
Year(s) Of Engagement Activity | 2017 |
Description | Presented ABC as part on NERC short course on Agent-based modelling organised by Bournemouth University Jan 28 2016 |
Form Of Engagement Activity | A formal working group, expert panel or dialogue |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | Spread knowledge of ABC to postgrads who may use it in the PhD |
Year(s) Of Engagement Activity | 2016 |
Description | Presented ABC for IBM to Centre for Ecology and Hydrology (4th Sept 2014) |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Professional Practitioners |
Results and Impact | Sparked request for further one-to-one tuition led to collaborative research |
Year(s) Of Engagement Activity | 2014 |
Description | Presented ABC for IBM to members of Rutherford Appleton Lab (21st August 2014) |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Professional Practitioners |
Results and Impact | Presentation sparked interest none so far |
Year(s) Of Engagement Activity | 2014 |
Description | Presented ABC for IBM to staff in Syngenta (2nd Sept 2014), |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Request for follow up information Formal collaboration now started |
Year(s) Of Engagement Activity | 2014 |
Description | Workshop at the British Ecological Conference 2016 |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | 80 researchers attended to discuss the construction and evaluation of Individual Based Models using Approximate Bayesian Computation and other techniques |
Year(s) Of Engagement Activity | 2016 |