Experimental large scale data services and visualisation facilities for Earth Observation and model data.

Lead Research Organisation: Plymouth Marine Laboratory
Department Name: Remote Sensing Group

Abstract

Web, database, storage and processing servers to:
* Provide data services for massive EO datasets to a range of scientist users
- Data access, subsetting, querying and selection..
* Demonstrate and explore advanced online analytics and visualisation
- OGC services, processing (WPS), advanced filtering (WCPS)
* Establish a services exemplar and make it easier for others to create similar systems

Planned Impact

The data services will enable access to satellite and model data on a scale not previously possible. Freely available, they may be used by any organisation (academic or commercial), supporting work including model assimilation, rapid response for research cruises or environmental monitoring, climate change studies and policy recommendations. NERC, ESA and EC projects covering these topics are already underway at PML. Making such data sets available globally is likely to increase their uptake through network effects in the academic community. NOAA's Earth System Research Laboratory made atmospheric model outputs readily available and significantly increased their use in the modelling community.

As importantly, a recommended data services infrastructure structure with free availability of software, supported by a demonstration facility, simplifies and speeds up the deployment of similar services at other sites, including large central data archives and processing centres. The model suggested also allows simple integration of remote data sources into local processing or visualisation. For example, visualisation facilities at ISIC/CEMS could seamlessly pull data from PML.

Publications

10 25 50
 
Description The grant provided infrastructure that could support modern and novel "big data" data services in a robust fashion, with the primary purposes of allowing people without this infrastructure to supply data, to disseminate information on how to set up such infrastructure and to trailblaze new methods to explore whether they are worthwhile for the Earth observation, modelling and oceanography communities.
Exploitation Route 1. Deployment of their data on this cluster
2. Development of their own cluster following some of the instructions produced by this grant
3. Adoption (or justified rejection!) of new methods, based on exploratory work done with this infrastructure
4. Further development of methods, based on experiences with handling extremely large datasets with this infrastructure
Sectors Aerospace, Defence and Marine,Digital/Communication/Information Technologies (including Software),Environment

URL https://vortices.npm.ac.uk/
 
Description The data services have been used to supply data for a range of international projects, including FP7, ESA and UNEP funded ones, and a number of internal and external collaborators. We hope that a good experience with robust data services will lead to greater deployment of similar systems, which are essential to deal with modern data volumes and usage patterns. The data served has been used globally by a range of research communities (mainly Earth Observation, modelling and oceanography), but are also publically accessible to all. Exploration of the new methods has resulted in requests for advice from numerous centres across Europe. We have also organised a workshop on big data services, where our findings were disseminated and discussed with other interested parties (mainly ICT oriented) and have shared our key experiences with the GEO European Projects Workshop.
First Year Of Impact 2014
Sector Aerospace, Defence and Marine,Digital/Communication/Information Technologies (including Software),Environment
Impact Types Policy & public services

 
Description ESA Climate Change Initiative 
Organisation European Space Agency
Country France 
Sector Public 
PI Contribution Provision of data services for the Ocean Colour Climate Change Initiative project. Includes novel use of WCPS querying to exploit the availability of per pixel uncertainty estimates and water type classification embedded in the data.
Collaborator Contribution 15 year unified time series of SeaWiFS, MODIS and MERIS OC data.
Impact Normally these data are available only over FTP. For the size of this dataset, subsetting services like OPeNDAP are essential and were explicitly requested by end users, ranging from ecosystem modellers, Earth observation specialists to consultancies, from Japan to Europe to America. Primary data at https://rsg.pml.ac.uk/thredds/catalog-cci.html , portal at http://www.oceancolour.org/
Start Year 2014
 
Description Ecosystem modellers collaboration 
Organisation Plymouth Marine Laboratory
Country United Kingdom 
Sector Academic/University 
PI Contribution Internal collaboration with the modelling group within PML, which wanted to use their model outputs via OPeNDAP data services, but lacked the infrastructure and experience to establish them.
Collaborator Contribution Data are publically available and potentially to be used by other collaborators working on the open source FVCOM model.
Impact Additional freely available test data from a contemporary model.
Start Year 2014
 
Description FP7 Aqua-Users 
Organisation Dow Chemical Company (DOW)
Department Dow Haltermann
Country United Kingdom 
Sector Private 
PI Contribution Providing the user interface and underlying data services
Collaborator Contribution Data supply (freely available)
Impact From the big data cluster standpoint, the main output is the additional data availability and ongoing development work (plus sharing of experiences).
Start Year 2014
 
Description FP7 Aqua-Users 
Organisation Dow Chemical Company (DOW)
Department Dow Haltermann
Country United Kingdom 
Sector Private 
PI Contribution Providing the user interface and underlying data services
Collaborator Contribution Data supply (freely available)
Impact From the big data cluster standpoint, the main output is the additional data availability and ongoing development work (plus sharing of experiences).
Start Year 2014
 
Description FP7 Aqua-Users 
Organisation European Commission
Country European Union (EU) 
Sector Public 
PI Contribution Providing the user interface and underlying data services
Collaborator Contribution Data supply (freely available)
Impact From the big data cluster standpoint, the main output is the additional data availability and ongoing development work (plus sharing of experiences).
Start Year 2014
 
Description FP7 Aqua-Users 
Organisation Free University of Amsterdam
Department Department of Ecological Science
Country Netherlands 
Sector Academic/University 
PI Contribution Providing the user interface and underlying data services
Collaborator Contribution Data supply (freely available)
Impact From the big data cluster standpoint, the main output is the additional data availability and ongoing development work (plus sharing of experiences).
Start Year 2014
 
Description FP7 Aqua-Users 
Organisation Norwegian Institute for Water Research (NIVA)
Country Norway 
Sector Public 
PI Contribution Providing the user interface and underlying data services
Collaborator Contribution Data supply (freely available)
Impact From the big data cluster standpoint, the main output is the additional data availability and ongoing development work (plus sharing of experiences).
Start Year 2014
 
Description FP7 Aqua-Users 
Organisation University of Lisbon
Department Faculty of Science (FCUL)
Country Portugal 
Sector Academic/University 
PI Contribution Providing the user interface and underlying data services
Collaborator Contribution Data supply (freely available)
Impact From the big data cluster standpoint, the main output is the additional data availability and ongoing development work (plus sharing of experiences).
Start Year 2014
 
Description FP7 Aqua-Users 
Organisation Water Insight B.V.
Country Netherlands 
Sector Private 
PI Contribution Providing the user interface and underlying data services
Collaborator Contribution Data supply (freely available)
Impact From the big data cluster standpoint, the main output is the additional data availability and ongoing development work (plus sharing of experiences).
Start Year 2014
 
Description FP7 Earth2Observe 
Organisation Deltares
Country Netherlands 
Sector Charity/Non Profit 
PI Contribution These subset of partners needed to supply large volumes of data over WxS data services and lacked the capability. PML is hosting their data and assisting them to set up their own services, when desired.
Collaborator Contribution This project will produce a major reevaluation of the global water cycle; all of these data will be freely available.
Impact So far, a number of precursor datasets have been published; the reanalysis datasets are due to be published in the next year or two. The collaboration includes ICT specialists, modellers and earth observation scientists.
Start Year 2013
 
Description FP7 Earth2Observe 
Organisation European Centre for Medium Range Weather Forecasting ECMWF
Country European Union (EU) 
Sector Public 
PI Contribution These subset of partners needed to supply large volumes of data over WxS data services and lacked the capability. PML is hosting their data and assisting them to set up their own services, when desired.
Collaborator Contribution This project will produce a major reevaluation of the global water cycle; all of these data will be freely available.
Impact So far, a number of precursor datasets have been published; the reanalysis datasets are due to be published in the next year or two. The collaboration includes ICT specialists, modellers and earth observation scientists.
Start Year 2013
 
Description FP7 Earth2Observe 
Organisation European Commission
Country European Union (EU) 
Sector Public 
PI Contribution These subset of partners needed to supply large volumes of data over WxS data services and lacked the capability. PML is hosting their data and assisting them to set up their own services, when desired.
Collaborator Contribution This project will produce a major reevaluation of the global water cycle; all of these data will be freely available.
Impact So far, a number of precursor datasets have been published; the reanalysis datasets are due to be published in the next year or two. The collaboration includes ICT specialists, modellers and earth observation scientists.
Start Year 2013
 
Description FP7 Earth2Observe 
Organisation Free University of Amsterdam
Country Netherlands 
Sector Academic/University 
PI Contribution These subset of partners needed to supply large volumes of data over WxS data services and lacked the capability. PML is hosting their data and assisting them to set up their own services, when desired.
Collaborator Contribution This project will produce a major reevaluation of the global water cycle; all of these data will be freely available.
Impact So far, a number of precursor datasets have been published; the reanalysis datasets are due to be published in the next year or two. The collaboration includes ICT specialists, modellers and earth observation scientists.
Start Year 2013
 
Description FP7 Earth2Observe 
Organisation National Research Council
Country Italy 
Sector Public 
PI Contribution These subset of partners needed to supply large volumes of data over WxS data services and lacked the capability. PML is hosting their data and assisting them to set up their own services, when desired.
Collaborator Contribution This project will produce a major reevaluation of the global water cycle; all of these data will be freely available.
Impact So far, a number of precursor datasets have been published; the reanalysis datasets are due to be published in the next year or two. The collaboration includes ICT specialists, modellers and earth observation scientists.
Start Year 2013
 
Description FP7 Earth2Observe 
Organisation Vienna University of Technology
Country Austria 
Sector Academic/University 
PI Contribution These subset of partners needed to supply large volumes of data over WxS data services and lacked the capability. PML is hosting their data and assisting them to set up their own services, when desired.
Collaborator Contribution This project will produce a major reevaluation of the global water cycle; all of these data will be freely available.
Impact So far, a number of precursor datasets have been published; the reanalysis datasets are due to be published in the next year or two. The collaboration includes ICT specialists, modellers and earth observation scientists.
Start Year 2013
 
Description FP7 OPEC 
Organisation Centre For Environment, Fisheries And Aquaculture Science
Country United Kingdom 
Sector Public 
PI Contribution These partners in FP7 OPEC have specifically made use of the data services provided by the big data cluster
Collaborator Contribution The data they have contributed are now freely available.
Impact Major outputs are the contribution towards the OPEC project, which aims to develop and evaluate ecosystem forecast tools to help assess and manage the risks posed by human activities on the marine environment. The specific outputs from this collaboration are much faster and more reliable data services for the project and its users [primarily ecosystem modellers and earth observation specialists], and the greater availability of the datasets produced by the project [general public and any other interested users]. The data services support a portal - http://marine-opec.eu/portal.html
Start Year 2014
 
Description FP7 OPEC 
Organisation Danish Meteorological Institute (DMI)
Country Denmark 
Sector Public 
PI Contribution These partners in FP7 OPEC have specifically made use of the data services provided by the big data cluster
Collaborator Contribution The data they have contributed are now freely available.
Impact Major outputs are the contribution towards the OPEC project, which aims to develop and evaluate ecosystem forecast tools to help assess and manage the risks posed by human activities on the marine environment. The specific outputs from this collaboration are much faster and more reliable data services for the project and its users [primarily ecosystem modellers and earth observation specialists], and the greater availability of the datasets produced by the project [general public and any other interested users]. The data services support a portal - http://marine-opec.eu/portal.html
Start Year 2014
 
Description FP7 OPEC 
Organisation European Commission
Country European Union (EU) 
Sector Public 
PI Contribution These partners in FP7 OPEC have specifically made use of the data services provided by the big data cluster
Collaborator Contribution The data they have contributed are now freely available.
Impact Major outputs are the contribution towards the OPEC project, which aims to develop and evaluate ecosystem forecast tools to help assess and manage the risks posed by human activities on the marine environment. The specific outputs from this collaboration are much faster and more reliable data services for the project and its users [primarily ecosystem modellers and earth observation specialists], and the greater availability of the datasets produced by the project [general public and any other interested users]. The data services support a portal - http://marine-opec.eu/portal.html
Start Year 2014
 
Description FP7 OPEC 
Organisation Italian National Institute of Oceanography and Experimental Geophysics (OGS)
Country Italy 
Sector Public 
PI Contribution These partners in FP7 OPEC have specifically made use of the data services provided by the big data cluster
Collaborator Contribution The data they have contributed are now freely available.
Impact Major outputs are the contribution towards the OPEC project, which aims to develop and evaluate ecosystem forecast tools to help assess and manage the risks posed by human activities on the marine environment. The specific outputs from this collaboration are much faster and more reliable data services for the project and its users [primarily ecosystem modellers and earth observation specialists], and the greater availability of the datasets produced by the project [general public and any other interested users]. The data services support a portal - http://marine-opec.eu/portal.html
Start Year 2014
 
Description FP7 OPEC 
Organisation Middle East Technical University
Department Institute of Marine Sciences
Country Turkey 
Sector Academic/University 
PI Contribution These partners in FP7 OPEC have specifically made use of the data services provided by the big data cluster
Collaborator Contribution The data they have contributed are now freely available.
Impact Major outputs are the contribution towards the OPEC project, which aims to develop and evaluate ecosystem forecast tools to help assess and manage the risks posed by human activities on the marine environment. The specific outputs from this collaboration are much faster and more reliable data services for the project and its users [primarily ecosystem modellers and earth observation specialists], and the greater availability of the datasets produced by the project [general public and any other interested users]. The data services support a portal - http://marine-opec.eu/portal.html
Start Year 2014
 
Description FP7 OPEC 
Organisation Plymouth Marine Laboratory
Country United Kingdom 
Sector Academic/University 
PI Contribution These partners in FP7 OPEC have specifically made use of the data services provided by the big data cluster
Collaborator Contribution The data they have contributed are now freely available.
Impact Major outputs are the contribution towards the OPEC project, which aims to develop and evaluate ecosystem forecast tools to help assess and manage the risks posed by human activities on the marine environment. The specific outputs from this collaboration are much faster and more reliable data services for the project and its users [primarily ecosystem modellers and earth observation specialists], and the greater availability of the datasets produced by the project [general public and any other interested users]. The data services support a portal - http://marine-opec.eu/portal.html
Start Year 2014
 
Description FP7 OPEC 
Organisation Technical University of Denmark
Country Denmark 
Sector Academic/University 
PI Contribution These partners in FP7 OPEC have specifically made use of the data services provided by the big data cluster
Collaborator Contribution The data they have contributed are now freely available.
Impact Major outputs are the contribution towards the OPEC project, which aims to develop and evaluate ecosystem forecast tools to help assess and manage the risks posed by human activities on the marine environment. The specific outputs from this collaboration are much faster and more reliable data services for the project and its users [primarily ecosystem modellers and earth observation specialists], and the greater availability of the datasets produced by the project [general public and any other interested users]. The data services support a portal - http://marine-opec.eu/portal.html
Start Year 2014
 
Description NCOF 
Organisation Meteorological Office UK
Country United Kingdom 
Sector Public 
PI Contribution Provision of data services; the Met Office was prevented from doing so due to on-site security restrictions
Collaborator Contribution Public availability of the MRCS model outputs for the UK area, then superseded by FOAM.
Impact Public availability of the MRCS model outputs for the UK area, then superseded by FOAM. Data feed at http://ncof.pml.ac.uk/
Start Year 2008
 
Description NCOF 
Organisation The National Centre for Ocean Forecasting
Country United Kingdom 
Sector Public 
PI Contribution Provision of data services; the Met Office was prevented from doing so due to on-site security restrictions
Collaborator Contribution Public availability of the MRCS model outputs for the UK area, then superseded by FOAM.
Impact Public availability of the MRCS model outputs for the UK area, then superseded by FOAM. Data feed at http://ncof.pml.ac.uk/
Start Year 2008
 
Description NEODAAS 
Organisation Plymouth Marine Laboratory
Country United Kingdom 
Sector Academic/University 
PI Contribution Experimental data services for specific areas of interest. NEODAAS has had a long term interest in web services, but never fully deployed them. This test case may help tip the balance.
Collaborator Contribution Availability of a ~20 year satellite time series by new access methods, albeit only in small parts. The intention is to make the full series available, if technically possible.
Impact Availability of a ~20 year satellite time series by new access methods, albeit only in small parts. The intention is to make the full series available, if technically possible. Collaboration between computing specialists and earth observation scientists.
Start Year 2006
 
Description Transboundary Waters Assessment Programme data services 
Organisation Plymouth Marine Laboratory
Country United Kingdom 
Sector Academic/University 
PI Contribution The project wanted to add its data outputs to some websites and portals, which required WxS data services. The team did not have the capability or infrastructure themselves.
Collaborator Contribution Additional public data.
Impact Additional public data, e.g. http://twapgeoportal.grid.unep.ch/geonetwork/srv/eng/metadata.show?id=1141&currTab=simple and https://rsg.pml.ac.uk/thredds/catalog-twap.html Collaboration includes oceanographers / remote sensing scientists and computing.
Start Year 2014
 
Description Transboundary Waters Assessment Programme data services 
Organisation United Nations (UN)
Department United Nations Environment Programme
Country Kenya 
Sector Charity/Non Profit 
PI Contribution The project wanted to add its data outputs to some websites and portals, which required WxS data services. The team did not have the capability or infrastructure themselves.
Collaborator Contribution Additional public data.
Impact Additional public data, e.g. http://twapgeoportal.grid.unep.ch/geonetwork/srv/eng/metadata.show?id=1141&currTab=simple and https://rsg.pml.ac.uk/thredds/catalog-twap.html Collaboration includes oceanographers / remote sensing scientists and computing.
Start Year 2014
 
Description Western Channel Observatory 
Organisation Plymouth Marine Laboratory
Country United Kingdom 
Sector Academic/University 
PI Contribution Provision of WxS data services and integration into portals for WCO live outputs, which the WCO group did not have the capability to do themselves.
Collaborator Contribution Greater availability of near-real-time insitu data from the WC Observatory
Impact A number of data streams are potentially available. The most recent is from the new Penlee on-shore observation station - https://rsg.pml.ac.uk/thredds/catalog-penlee.html
Start Year 2008
 
Title Big data cluster 
Description The purpose of this grant was to fund infrastructure to provide reliable and robust data and processing services, and particularly to explore the technology in the context of very large datasets, with the aim of improving the technology, the variety and number of deployments and the knowledge in the community about these services / how to set them up. 
Type Of Technology New/Improved Technique/Technology 
Year Produced 2014 
Impact Significant numbers of large datasets made available to an actively global userbase. Dissemination of recommendations and key setup information to a range of organisations, from ICT specialists to scientific organisations wanting similar systems but lacking the experience. Exploration of cutting-edge technologies for handling extremely large datasets, both on a service level and as a user experience, plus dissemination of these experiences. 
URL https://vortices.npm.ac.uk/