Models in the Cloud: Generative Software Frameworks to Support the Execution of Environmental Models in the Cloud

Lead Research Organisation: Lancaster University
Department Name: Computing & Communications

Abstract

Cloud computing is one of the most important technological developments of the last few years, with the technology having a major and transformative impact on many areas of society and the economy. For example, cloud computing underpins developments in smart cities, ecommerce and eGovernment and also provides the storage and computational capabilities that underpin data science (cf. big data).
The software support offered by cloud computing is however in its infancy and is tailored towards particular styles of application. This is particularly true with respect to computation in the cloud, where the programming approach offered by the computational framework MapReduce dominates and yet MapReduce assumes a particular style of programming where potentially massive data sets are analysed by a map() operation before results are collated through an associated reduce() operation. This is powerful but very limited.
In parallel, researchers are also interested in realising the benefits of cloud computing in many other areas of application. This project focuses on the support offered by cloud computing to Environmental Science and, in particular, to the execution of potentially complex environmental models in the cloud. This is an area of huge significance, with environmental modelling being the key tool to evaluate uncertainty, risk assessment, and mitigation strategies around flood/ drought, food security and the impact of climate change (with major consequences for the economy and for society). We particularly focus on the principles and techniques in the key areas of Platform as a Service (or PaaS), effectively the middleware for cloud computing. The central insight is that services at the PaaS level need to be more carefully tailored to the needs of key application domains, including but not limited to support for the execution of complex environmental models. We advocate a novel approach based on a combination of model-driven engineering coupled with software frameworks and argue that this enables a paradigm shift in terms of the flexible and tailored support offered by cloud computing for given application domains.
Key beneficiaries of this work include the computer science communities working on model-driven engineering and cloud computing, researchers from environmental modelling in areas ranging from climate change modellers to flood prediction, and also key stakeholders related to environmental management and we include an exciting range of partner institutions from this area to maximise the impact of this work.

Planned Impact

The proposed research is well balanced between cutting edge research and impact on society and the economy. In terms of the latter, we seek a transformational impact on the target scientific community and also on users of environmental models who deal with uncertainty, risk assessment, and mitigation strategies around flood/ drought, food security and climate change. We adopt a multi-faceted impact strategy around the following strands (discussed in more detail below):

1. The establishment of an Experts Group as our key vehicle for end user engagement and dissemination and exploitation (new insights and approaches);
2. Working with appropriate bodies to seek a transformative impact on how Environmental Science more generally is carried out (new paradigms);
3. Working with those involved in big data initiatives in the UK to enhance infrastructure capabilities for environmental modelling (new tools);
4. Working with another research institute (EBTIC) to generalise the results beyond Environmental Science (new avenues).

This is supported by a series of strategic partners, and their their roles are summarised below.

Centre for Ecology and Hydrology (CEH): bringing domain knowledge around environmental modelling and access to NERC cloud infrastructure; supporting strands 1, 2, 3.
Community Earth System Model (CESM) via NCAR: bringing domain knowledge around environmental modelling (climate change); supporting strands 1, 2.
JBA Trust: bringing domain knowledge and application of environmental modelling to a range of real world problem domains; supporting strands 1, 2.
UK Earth System Model (UKESM) via Met Office/ NERC: bringing domain knowledge around environmental modelling (climate change); supporting strands 1, 2.
EBTIC: bringing domain knowledge around next generation networks, MDE expertise, generalisation of results; supporting strand 4.

As well as these impact strands, we also plan public engagement activities which we believe are important in scientific research (will include public lectures and Café Scientifique events).

Publications

10 25 50
 
Description We have deep experience of how to run potentially complex environmental models in the cloud, including working with WRF and an eDNA analysis pipeline. It is increasingly important for environmental science models to take advantage of contemporary cloud infrastructure and this has added benefits in terms of democratising access to often complex models an dis also directly supportive of a more open, collaborative and transparent style of science. Having models as a service in the cloud reduced significantly the effort of using these models allowing scientists to focus on science rather than complexities associated with getting the model to run.

Model execution in the cloud is an important prerequisite for digital twins and results from this research are feeding into NERC strategy around digital twins for the natural environment.
Exploitation Route In using modelling as a service or doing integrated modelling in a cloud environment - by members of the environmental sciences communities.

As a prerequisite for digital twins of the natural environment - as discussed above.
Sectors Digital/Communication/Information Technologies (including Software),Environment

URL https://www.ensembleprojects.org/
 
Description Input into architectures and approaches to digital twins - which are seen as important tools for government, policy makers and so on.
First Year Of Impact 2022
Sector Digital/Communication/Information Technologies (including Software),Environment
Impact Types Societal,Economic,Policy & public services

 
Description DAFNI (Data & Analytics Facility for National Infrastructure) mid-year review (as member of expert review board) - DAFNI is an EPSRC funded initiative intended to develop a national platform to satisfy the computational needs in support of data analysis; infrastructure research; and strategic thinking for the UK's long term planning and investment needs.
Geographic Reach National 
Policy Influence Type Membership of a guideline committee
URL https://www.dafni.ac.uk/
 
Description IGAC Conference - Paul Young
Geographic Reach Multiple continents/international 
Policy Influence Type Influenced training of practitioners or researchers
Impact |Paul Young attended the IGAC Conference inTakamatsu, where he talked to a global community of atmospheric chemists. He discussed exploiting the Models in the Cloud concepts with a wide range of people, particularly a growing group of atmospheric scientists based in Africa. The project's developments are particularly relevant for that group, given the comparative lack of high performance computing resources.
 
Description SPARC General Assembly - Paul Young
Geographic Reach Multiple continents/international 
Policy Influence Type Influenced training of practitioners or researchers
Impact Paul Young attended the SPARC General Assembly in Kyoto, where he was talking to a global community of climate scientists and policy experts. Discussions with a global community of atmospheric chemists on exploiting the Models in the Cloud concepts, with an emphasis on how MIC tools could be used in a decision making context.
 
Description WMO Global Atmospheric Watch - Paul Young
Geographic Reach Multiple continents/international 
Policy Influence Type Influenced training of practitioners or researchers
Impact Paul Young attended the WMO Global Atmospheric Watch in Osaka, where he discussed how MIC concepts and outputs could help measurement scientists use sophisticated models to make sense of their data.
 
Description ABC: Adaptive Brokerage for the Cloud, funded by the EPSRC (Responsive Mode)
Amount £388,701 (GBP)
Funding ID EP/R010889/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 01/2018 
End 12/2020
 
Description AI 4 Earth grant for Azure
Amount $15,000 (USD)
Organisation Microsoft Research 
Sector Private
Country Global
Start 02/2019 
End 12/2019
 
Description CROP-Net: Monitoring and Predicting the Effects of Climate Change on Crop Yields
Amount £27,753 (GBP)
Funding ID NE/5017208/1 
Organisation Natural Environment Research Council 
Sector Public
Country United Kingdom
Start 02/2019 
End 01/2020
 
Description Data Science of the Natural Environment, New Approaches to Data Science EPSRC Call
Amount £2,656,400 (GBP)
Funding ID EP/R01860X/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 04/2018 
End 03/2023
 
Description Design Principles and Responsible Innovation for a Sustainable Digital Economy (Paris-DE)
Amount £895,718 (GBP)
Funding ID EP/V042378/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 08/2021 
End 08/2024
 
Description Microsoft AI for Earth grant
Amount $15,000 (USD)
Organisation Microsoft Research 
Sector Private
Country Global
Start 02/2019 
End 02/2020
 
Description Monitoring and predicting the effects of climate change on crop yields
Amount £27,752 (GBP)
Funding ID NE/S017208/1 
Organisation Natural Environment Research Council 
Sector Public
Country United Kingdom
Start 02/2019 
End 01/2020
 
Title Data: Ensemble simulation using the Weather Research & Forecasting (WRF) model 
Description Data from the paper currently in review titled: A large ensemble approach to quantifying internal model variability within the Weather Research & Forecasting (WRF) numerical model.The dataset contains (i) the configuration files required to re-run the ensemble and (ii) model outputs for the 244 member ensemble. This data set currently has a 6 month embargo (until 1st July 2020) 
Type Of Material Database/Collection of data 
Year Produced 2020 
Provided To Others? No  
Impact Not aware of any yet 
 
Description Abstract: A large ensemble approach to quantifying internal model variability in the Weather Research & Forecasting (WRF) model 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Richard Bassett presented the abstract at AGU Fall meeting on 13 December 2019.
Year(s) Of Engagement Activity 2019
URL https://agu.confex.com/agu/fm19/meetingapp.cgi/Paper/496290
 
Description CMWR Conference - Opportunities for software... 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Richard Bassett presented 'Opportunities for software engineering practices in deploying environmental models to cloud computing architectures' at the CMWR Conference to a mainly academic audience.
Year(s) Of Engagement Activity 2018
 
Description Conference paper: A software framework to support collaborative and reproducible scientific experiments on the cloud 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Presented by Will Simm at AGU fall meeting
Year(s) Of Engagement Activity 2019
 
Description EGU General Assembly - Models of Models.. 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Will Simm presented 'Models of Models!? Using Domain Specific Languages in Environmental Modelling', at the EGU General Assembly to a mainly academic audience.
Year(s) Of Engagement Activity 2018
 
Description EGU General Assembly - Pie in the Sky.. 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Study participants or study members
Results and Impact Will Simm presented 'Pie in the sky or models in the cloud? Unlocking cloud architectures for environmental modelling with Model Driven Engineering' at the EGU General Assembly to a mainly academic audience.
Year(s) Of Engagement Activity 2018
 
Description EMS Conference - To InfiniBand.. 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Richard Bassett presented 'To InfiniBand, and beyond? Cloud computing for the WRF model, at the EMS Conference to a mainly academic audience.
Year(s) Of Engagement Activity 2018
 
Description IGAC Conference 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Paul Young in Takamatsu, talking to a global community of atmospheric chemists. Discussed exploiting the Models in the Cloud concepts with a wide range of people, particularly a growing group of atmospheric scientists based in Africa. The project's developments are particularly relevant for that group, given the comparative lack of high performance computing resources.

Outcomes and impacts: Engagement with colleagues running the WRFChem model. Possible side-/follow on project discussed exploiting MIC advances for WRF in this similar model.
Year(s) Of Engagement Activity 2018
 
Description Poster Presentation: Rapid urbanisation in Lagos and the growth of a mega-city urban heat island 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Presented by Richard Bassett at AGU FAll meeting on 11th December 2019
Year(s) Of Engagement Activity 2019
URL https://agu.confex.com/agu/fm19/meetingapp.cgi/Paper/514898
 
Description Poster presentation at AGU: Bringing models to end users: Visioning the new end-to-end data science and compute tools required to build resilience in the context of a changing climate. 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Poster presentation at AGU Fall 2019, by Will Simm
Year(s) Of Engagement Activity 2019
 
Description SPARC General Assembly 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Paul Young visited Kyoto, talking to a global community of climate scientists and policy experts. Discussions with participants around exploiting MiC, but with an emphasis on how MIC tools could be used in a decision making context (which is the focus of the project's next stage).
Year(s) Of Engagement Activity 2017,2018
 
Description Team visit to Rennes to meet at INRIA 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact 2 day visit to Rennes to learn about state of the art Model Driven Engineering tools. As a result of the meeting a workshop has been planned at CMWR in St Malo in June titled 'Application of Advanced Software Engineering Tools and Methods in the Environmental Sciences'. We will be organising this and presenting two papers.
Year(s) Of Engagement Activity 2017
URL http://cmwrconference.org/program/sessions/
 
Description Visit to JRC, Ispra 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Prof. Blair presented the work of the Fellowship and also Models in the Cloud to the EU's Joint Research Centre, in Ispra Italy, disseminating results of this work and also building partnerships for ongoing collaboration. This is important as JRC are at the heart of work to influence European policy based on scientific evidence. The visit took place between 5th and 7th July 2017.
Year(s) Of Engagement Activity 2017
 
Description Visit to thr Informatics Lab, Met Office 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact The Ensemble group visited the Met Office, Informatics Lab on 28th November 2018 to discuss common interests around digital technology and environmental change, and to discuss future collaborations, for example around the concept of virtual labs. The Informatics Lab (https://www.informaticslab.co.uk/) has a mission to rapidly develop prototypes which explore how we can use new technology, science and design to make our data useful, and hence there is a significant overlap with the work in Ensemble.
Year(s) Of Engagement Activity 2018
URL https://www.ensembleprojects.org/667-2/
 
Description WMO Global Atmospheric Watch 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Policymakers/politicians
Results and Impact Paul Young at Osaka, where he discussed how MIC concepts and outputs could help measurement scientists use sophisticated models to make sense of their data.
Year(s) Of Engagement Activity 2018
 
Description Workshop on Software Engineering for Sustainability 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact A two day workshop was held involving members of Lancaster University and INRIA, held in Rennes on 28th February - 1st March 2017. The purpose was to share our research on the topic of software engineering and sustainability/ living with environmental change, and to look at specific collaboration around the role of model-driven engineering in supporting such work.
This workshop builds on a previous visit by Benoit Combemale to Lancaster University on 5th and 6th December 2016.
It is anticipated that this will be an important ongoing collaboration for the Models in the Cloud project and the greater Living with Environmental Change activities at Lancaster.
Year(s) Of Engagement Activity 2017