Models in the Cloud: Generative Software Frameworks to Support the Execution of Environmental Models in the Cloud
Lead Research Organisation:
Lancaster University
Department Name: Computing & Communications
Abstract
Cloud computing is one of the most important technological developments of the last few years, with the technology having a major and transformative impact on many areas of society and the economy. For example, cloud computing underpins developments in smart cities, ecommerce and eGovernment and also provides the storage and computational capabilities that underpin data science (cf. big data).
The software support offered by cloud computing is however in its infancy and is tailored towards particular styles of application. This is particularly true with respect to computation in the cloud, where the programming approach offered by the computational framework MapReduce dominates and yet MapReduce assumes a particular style of programming where potentially massive data sets are analysed by a map() operation before results are collated through an associated reduce() operation. This is powerful but very limited.
In parallel, researchers are also interested in realising the benefits of cloud computing in many other areas of application. This project focuses on the support offered by cloud computing to Environmental Science and, in particular, to the execution of potentially complex environmental models in the cloud. This is an area of huge significance, with environmental modelling being the key tool to evaluate uncertainty, risk assessment, and mitigation strategies around flood/ drought, food security and the impact of climate change (with major consequences for the economy and for society). We particularly focus on the principles and techniques in the key areas of Platform as a Service (or PaaS), effectively the middleware for cloud computing. The central insight is that services at the PaaS level need to be more carefully tailored to the needs of key application domains, including but not limited to support for the execution of complex environmental models. We advocate a novel approach based on a combination of model-driven engineering coupled with software frameworks and argue that this enables a paradigm shift in terms of the flexible and tailored support offered by cloud computing for given application domains.
Key beneficiaries of this work include the computer science communities working on model-driven engineering and cloud computing, researchers from environmental modelling in areas ranging from climate change modellers to flood prediction, and also key stakeholders related to environmental management and we include an exciting range of partner institutions from this area to maximise the impact of this work.
The software support offered by cloud computing is however in its infancy and is tailored towards particular styles of application. This is particularly true with respect to computation in the cloud, where the programming approach offered by the computational framework MapReduce dominates and yet MapReduce assumes a particular style of programming where potentially massive data sets are analysed by a map() operation before results are collated through an associated reduce() operation. This is powerful but very limited.
In parallel, researchers are also interested in realising the benefits of cloud computing in many other areas of application. This project focuses on the support offered by cloud computing to Environmental Science and, in particular, to the execution of potentially complex environmental models in the cloud. This is an area of huge significance, with environmental modelling being the key tool to evaluate uncertainty, risk assessment, and mitigation strategies around flood/ drought, food security and the impact of climate change (with major consequences for the economy and for society). We particularly focus on the principles and techniques in the key areas of Platform as a Service (or PaaS), effectively the middleware for cloud computing. The central insight is that services at the PaaS level need to be more carefully tailored to the needs of key application domains, including but not limited to support for the execution of complex environmental models. We advocate a novel approach based on a combination of model-driven engineering coupled with software frameworks and argue that this enables a paradigm shift in terms of the flexible and tailored support offered by cloud computing for given application domains.
Key beneficiaries of this work include the computer science communities working on model-driven engineering and cloud computing, researchers from environmental modelling in areas ranging from climate change modellers to flood prediction, and also key stakeholders related to environmental management and we include an exciting range of partner institutions from this area to maximise the impact of this work.
Planned Impact
The proposed research is well balanced between cutting edge research and impact on society and the economy. In terms of the latter, we seek a transformational impact on the target scientific community and also on users of environmental models who deal with uncertainty, risk assessment, and mitigation strategies around flood/ drought, food security and climate change. We adopt a multi-faceted impact strategy around the following strands (discussed in more detail below):
1. The establishment of an Experts Group as our key vehicle for end user engagement and dissemination and exploitation (new insights and approaches);
2. Working with appropriate bodies to seek a transformative impact on how Environmental Science more generally is carried out (new paradigms);
3. Working with those involved in big data initiatives in the UK to enhance infrastructure capabilities for environmental modelling (new tools);
4. Working with another research institute (EBTIC) to generalise the results beyond Environmental Science (new avenues).
This is supported by a series of strategic partners, and their their roles are summarised below.
Centre for Ecology and Hydrology (CEH): bringing domain knowledge around environmental modelling and access to NERC cloud infrastructure; supporting strands 1, 2, 3.
Community Earth System Model (CESM) via NCAR: bringing domain knowledge around environmental modelling (climate change); supporting strands 1, 2.
JBA Trust: bringing domain knowledge and application of environmental modelling to a range of real world problem domains; supporting strands 1, 2.
UK Earth System Model (UKESM) via Met Office/ NERC: bringing domain knowledge around environmental modelling (climate change); supporting strands 1, 2.
EBTIC: bringing domain knowledge around next generation networks, MDE expertise, generalisation of results; supporting strand 4.
As well as these impact strands, we also plan public engagement activities which we believe are important in scientific research (will include public lectures and Café Scientifique events).
1. The establishment of an Experts Group as our key vehicle for end user engagement and dissemination and exploitation (new insights and approaches);
2. Working with appropriate bodies to seek a transformative impact on how Environmental Science more generally is carried out (new paradigms);
3. Working with those involved in big data initiatives in the UK to enhance infrastructure capabilities for environmental modelling (new tools);
4. Working with another research institute (EBTIC) to generalise the results beyond Environmental Science (new avenues).
This is supported by a series of strategic partners, and their their roles are summarised below.
Centre for Ecology and Hydrology (CEH): bringing domain knowledge around environmental modelling and access to NERC cloud infrastructure; supporting strands 1, 2, 3.
Community Earth System Model (CESM) via NCAR: bringing domain knowledge around environmental modelling (climate change); supporting strands 1, 2.
JBA Trust: bringing domain knowledge and application of environmental modelling to a range of real world problem domains; supporting strands 1, 2.
UK Earth System Model (UKESM) via Met Office/ NERC: bringing domain knowledge around environmental modelling (climate change); supporting strands 1, 2.
EBTIC: bringing domain knowledge around next generation networks, MDE expertise, generalisation of results; supporting strand 4.
As well as these impact strands, we also plan public engagement activities which we believe are important in scientific research (will include public lectures and Café Scientifique events).
Publications
Bassett R
(2020)
A Large Ensemble Approach to Quantifying Internal Model Variability Within the WRF Numerical Model
in Journal of Geophysical Research: Atmospheres
Blair G
(2019)
Data Science of the Natural Environment: A Research Roadmap
in Frontiers in Environmental Science
Blair GS
(2021)
Digital twins of the natural environment.
in Patterns (New York, N.Y.)
Simm W
(2018)
Extended Abstract: Pie in the sky or models in the cloud? Unlocking cloud architectures for environmental modelling with Model Driven Engineering.
in Geophysical Research Abstracts
Blair G
(2019)
Modeling for Sustainability (Dagstuhl Seminar 18351)
in Dagstuhl Reports (DagRep)
Blair G
(2019)
Models of everywhere revisited: A technological perspective
in Environmental Modelling & Software
Simm W
(2018)
SE in ES
Hollaway MJ
(2020)
Tackling the Challenges of 21st-Century Open Science and Beyond: A Data Science Lab Approach.
in Patterns (New York, N.Y.)
Bassett R
(2020)
The Megacity Lagos and Three Decades of Urban Heat Island Growth
in Journal of Applied Meteorology and Climatology
Blair GS
(2021)
The Role of Digital Technologies in Responding to the Grand Challenges of the Natural Environment: The Windermere Accord.
in Patterns (New York, N.Y.)
Samreen F
(2022)
Transferable Knowledge for Low-Cost Decision Making in Cloud Environments
in IEEE Transactions on Cloud Computing
Young P
(2018)
Tropospheric Ozone Assessment Report: Assessment of global-scale model performance for global and regional ozone distributions, variability, and trends
in Elementa: Science of the Anthropocene
Bassett R
(2020)
Urbanisation's contribution to climate warming in Great Britain
in Environmental Research Letters
Description | We have deep experience of how to run potentially complex environmental models in the cloud, including working with WRF and an eDNA analysis pipeline. It is increasingly important for environmental science models to take advantage of contemporary cloud infrastructure and this has added benefits in terms of democratising access to often complex models an dis also directly supportive of a more open, collaborative and transparent style of science. Having models as a service in the cloud reduced significantly the effort of using these models allowing scientists to focus on science rather than complexities associated with getting the model to run. Model execution in the cloud is an important prerequisite for digital twins and results from this research are feeding into NERC strategy around digital twins for the natural environment. |
Exploitation Route | In using modelling as a service or doing integrated modelling in a cloud environment - by members of the environmental sciences communities. As a prerequisite for digital twins of the natural environment - as discussed above. |
Sectors | Digital/Communication/Information Technologies (including Software) Environment |
URL | https://www.ensembleprojects.org/ |
Description | The findings from this grant have fed into architectures and approaches to digital twins - which are seen as important tools for government, policy makers and so on. More specifically, this work directly influenced the work on an Information Management Framework for Environmental Digital Twins in teh IMFe and follow-up P-IMFe projects funded by NERC. The work on models into teh cloud is now feeding into the digital strategy of the UK Centre of Ecology & Hydrology (UKCEH) and their work on future modelling approaches. |
Sector | Digital/Communication/Information Technologies (including Software),Environment |
Impact Types | Societal Economic Policy & public services |
Description | DAFNI (Data & Analytics Facility for National Infrastructure) mid-year review (as member of expert review board) - DAFNI is an EPSRC funded initiative intended to develop a national platform to satisfy the computational needs in support of data analysis; infrastructure research; and strategic thinking for the UK's long term planning and investment needs. |
Geographic Reach | National |
Policy Influence Type | Membership of a guideline committee |
URL | https://www.dafni.ac.uk/ |
Description | IGAC Conference - Paul Young |
Geographic Reach | Multiple continents/international |
Policy Influence Type | Influenced training of practitioners or researchers |
Impact | |Paul Young attended the IGAC Conference inTakamatsu, where he talked to a global community of atmospheric chemists. He discussed exploiting the Models in the Cloud concepts with a wide range of people, particularly a growing group of atmospheric scientists based in Africa. The project's developments are particularly relevant for that group, given the comparative lack of high performance computing resources. |
Description | SPARC General Assembly - Paul Young |
Geographic Reach | Multiple continents/international |
Policy Influence Type | Influenced training of practitioners or researchers |
Impact | Paul Young attended the SPARC General Assembly in Kyoto, where he was talking to a global community of climate scientists and policy experts. Discussions with a global community of atmospheric chemists on exploiting the Models in the Cloud concepts, with an emphasis on how MIC tools could be used in a decision making context. |
Description | WMO Global Atmospheric Watch - Paul Young |
Geographic Reach | Multiple continents/international |
Policy Influence Type | Influenced training of practitioners or researchers |
Impact | Paul Young attended the WMO Global Atmospheric Watch in Osaka, where he discussed how MIC concepts and outputs could help measurement scientists use sophisticated models to make sense of their data. |
Description | ABC: Adaptive Brokerage for the Cloud, funded by the EPSRC (Responsive Mode) |
Amount | £388,701 (GBP) |
Funding ID | EP/R010889/1 |
Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
Sector | Public |
Country | United Kingdom |
Start | 01/2018 |
End | 12/2020 |
Description | AI 4 Earth grant for Azure |
Amount | $15,000 (USD) |
Organisation | Microsoft Research |
Sector | Private |
Country | Global |
Start | 02/2019 |
End | 12/2019 |
Description | CROP-Net: Monitoring and Predicting the Effects of Climate Change on Crop Yields |
Amount | £27,753 (GBP) |
Funding ID | NE/5017208/1 |
Organisation | Natural Environment Research Council |
Sector | Public |
Country | United Kingdom |
Start | 02/2019 |
End | 01/2020 |
Description | Data Science of the Natural Environment, New Approaches to Data Science EPSRC Call |
Amount | £2,656,400 (GBP) |
Funding ID | EP/R01860X/1 |
Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
Sector | Public |
Country | United Kingdom |
Start | 03/2018 |
End | 03/2023 |
Description | Design Principles and Responsible Innovation for a Sustainable Digital Economy (Paris-DE) |
Amount | £895,718 (GBP) |
Funding ID | EP/V042378/1 |
Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
Sector | Public |
Country | United Kingdom |
Start | 08/2021 |
End | 08/2024 |
Description | Microsoft AI for Earth grant |
Amount | $15,000 (USD) |
Organisation | Microsoft Research |
Sector | Private |
Country | Global |
Start | 02/2019 |
End | 02/2020 |
Description | Monitoring and predicting the effects of climate change on crop yields |
Amount | £27,752 (GBP) |
Funding ID | NE/S017208/1 |
Organisation | Natural Environment Research Council |
Sector | Public |
Country | United Kingdom |
Start | 02/2019 |
End | 01/2020 |
Title | Data: Ensemble simulation using the Weather Research & Forecasting (WRF) model |
Description | Data from the paper currently in review titled: A large ensemble approach to quantifying internal model variability within the Weather Research & Forecasting (WRF) numerical model.The dataset contains (i) the configuration files required to re-run the ensemble and (ii) model outputs for the 244 member ensemble. This data set currently has a 6 month embargo (until 1st July 2020) |
Type Of Material | Database/Collection of data |
Year Produced | 2020 |
Provided To Others? | No |
Impact | Not aware of any yet |
Description | Abstract: A large ensemble approach to quantifying internal model variability in the Weather Research & Forecasting (WRF) model |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Other audiences |
Results and Impact | Richard Bassett presented the abstract at AGU Fall meeting on 13 December 2019. |
Year(s) Of Engagement Activity | 2019 |
URL | https://agu.confex.com/agu/fm19/meetingapp.cgi/Paper/496290 |
Description | CMWR Conference - Opportunities for software... |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Richard Bassett presented 'Opportunities for software engineering practices in deploying environmental models to cloud computing architectures' at the CMWR Conference to a mainly academic audience. |
Year(s) Of Engagement Activity | 2018 |
Description | Conference paper: A software framework to support collaborative and reproducible scientific experiments on the cloud |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Other audiences |
Results and Impact | Presented by Will Simm at AGU fall meeting |
Year(s) Of Engagement Activity | 2019 |
Description | EGU General Assembly - Models of Models.. |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Will Simm presented 'Models of Models!? Using Domain Specific Languages in Environmental Modelling', at the EGU General Assembly to a mainly academic audience. |
Year(s) Of Engagement Activity | 2018 |
Description | EGU General Assembly - Pie in the Sky.. |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Study participants or study members |
Results and Impact | Will Simm presented 'Pie in the sky or models in the cloud? Unlocking cloud architectures for environmental modelling with Model Driven Engineering' at the EGU General Assembly to a mainly academic audience. |
Year(s) Of Engagement Activity | 2018 |
Description | EMS Conference - To InfiniBand.. |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Other audiences |
Results and Impact | Richard Bassett presented 'To InfiniBand, and beyond? Cloud computing for the WRF model, at the EMS Conference to a mainly academic audience. |
Year(s) Of Engagement Activity | 2018 |
Description | IGAC Conference |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Other audiences |
Results and Impact | Paul Young in Takamatsu, talking to a global community of atmospheric chemists. Discussed exploiting the Models in the Cloud concepts with a wide range of people, particularly a growing group of atmospheric scientists based in Africa. The project's developments are particularly relevant for that group, given the comparative lack of high performance computing resources. Outcomes and impacts: Engagement with colleagues running the WRFChem model. Possible side-/follow on project discussed exploiting MIC advances for WRF in this similar model. |
Year(s) Of Engagement Activity | 2018 |
Description | Poster Presentation: Rapid urbanisation in Lagos and the growth of a mega-city urban heat island |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Other audiences |
Results and Impact | Presented by Richard Bassett at AGU FAll meeting on 11th December 2019 |
Year(s) Of Engagement Activity | 2019 |
URL | https://agu.confex.com/agu/fm19/meetingapp.cgi/Paper/514898 |
Description | Poster presentation at AGU: Bringing models to end users: Visioning the new end-to-end data science and compute tools required to build resilience in the context of a changing climate. |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Other audiences |
Results and Impact | Poster presentation at AGU Fall 2019, by Will Simm |
Year(s) Of Engagement Activity | 2019 |
Description | SPARC General Assembly |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Other audiences |
Results and Impact | Paul Young visited Kyoto, talking to a global community of climate scientists and policy experts. Discussions with participants around exploiting MiC, but with an emphasis on how MIC tools could be used in a decision making context (which is the focus of the project's next stage). |
Year(s) Of Engagement Activity | 2017,2018 |
Description | Team visit to Rennes to meet at INRIA |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Postgraduate students |
Results and Impact | 2 day visit to Rennes to learn about state of the art Model Driven Engineering tools. As a result of the meeting a workshop has been planned at CMWR in St Malo in June titled 'Application of Advanced Software Engineering Tools and Methods in the Environmental Sciences'. We will be organising this and presenting two papers. |
Year(s) Of Engagement Activity | 2017 |
URL | http://cmwrconference.org/program/sessions/ |
Description | Visit to JRC, Ispra |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Prof. Blair presented the work of the Fellowship and also Models in the Cloud to the EU's Joint Research Centre, in Ispra Italy, disseminating results of this work and also building partnerships for ongoing collaboration. This is important as JRC are at the heart of work to influence European policy based on scientific evidence. The visit took place between 5th and 7th July 2017. |
Year(s) Of Engagement Activity | 2017 |
Description | Visit to thr Informatics Lab, Met Office |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | The Ensemble group visited the Met Office, Informatics Lab on 28th November 2018 to discuss common interests around digital technology and environmental change, and to discuss future collaborations, for example around the concept of virtual labs. The Informatics Lab (https://www.informaticslab.co.uk/) has a mission to rapidly develop prototypes which explore how we can use new technology, science and design to make our data useful, and hence there is a significant overlap with the work in Ensemble. |
Year(s) Of Engagement Activity | 2018 |
URL | https://www.ensembleprojects.org/667-2/ |
Description | WMO Global Atmospheric Watch |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Policymakers/politicians |
Results and Impact | Paul Young at Osaka, where he discussed how MIC concepts and outputs could help measurement scientists use sophisticated models to make sense of their data. |
Year(s) Of Engagement Activity | 2018 |
Description | Workshop on Software Engineering for Sustainability |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | A two day workshop was held involving members of Lancaster University and INRIA, held in Rennes on 28th February - 1st March 2017. The purpose was to share our research on the topic of software engineering and sustainability/ living with environmental change, and to look at specific collaboration around the role of model-driven engineering in supporting such work. This workshop builds on a previous visit by Benoit Combemale to Lancaster University on 5th and 6th December 2016. It is anticipated that this will be an important ongoing collaboration for the Models in the Cloud project and the greater Living with Environmental Change activities at Lancaster. |
Year(s) Of Engagement Activity | 2017 |