Experimental large scale data services and visualisation facilities for Earth Observation and model data.
Lead Research Organisation:
Plymouth Marine Laboratory
Department Name: Remote Sensing Group
Abstract
Web, database, storage and processing servers to:
* Provide data services for massive EO datasets to a range of scientist users
- Data access, subsetting, querying and selection..
* Demonstrate and explore advanced online analytics and visualisation
- OGC services, processing (WPS), advanced filtering (WCPS)
* Establish a services exemplar and make it easier for others to create similar systems
* Provide data services for massive EO datasets to a range of scientist users
- Data access, subsetting, querying and selection..
* Demonstrate and explore advanced online analytics and visualisation
- OGC services, processing (WPS), advanced filtering (WCPS)
* Establish a services exemplar and make it easier for others to create similar systems
Planned Impact
The data services will enable access to satellite and model data on a scale not previously possible. Freely available, they may be used by any organisation (academic or commercial), supporting work including model assimilation, rapid response for research cruises or environmental monitoring, climate change studies and policy recommendations. NERC, ESA and EC projects covering these topics are already underway at PML. Making such data sets available globally is likely to increase their uptake through network effects in the academic community. NOAA's Earth System Research Laboratory made atmospheric model outputs readily available and significantly increased their use in the modelling community.
As importantly, a recommended data services infrastructure structure with free availability of software, supported by a demonstration facility, simplifies and speeds up the deployment of similar services at other sites, including large central data archives and processing centres. The model suggested also allows simple integration of remote data sources into local processing or visualisation. For example, visualisation facilities at ISIC/CEMS could seamlessly pull data from PML.
As importantly, a recommended data services infrastructure structure with free availability of software, supported by a demonstration facility, simplifies and speeds up the deployment of similar services at other sites, including large central data archives and processing centres. The model suggested also allows simple integration of remote data sources into local processing or visualisation. For example, visualisation facilities at ISIC/CEMS could seamlessly pull data from PML.
Organisations
- Plymouth Marine Laboratory (Lead Research Organisation)
- Deltares (Collaboration)
- Danish Meteorological Institute (DMI) (Collaboration)
- United Nations (UN) (Collaboration)
- University of Lisbon (Collaboration)
- TU Wien (Collaboration)
- The National Centre for Ocean Forecasting (Collaboration)
- PLYMOUTH MARINE LABORATORY (Collaboration)
- European Centre for Medium Range Weather Forecasting ECMWF (Collaboration)
- Technical University of Denmark (Collaboration)
- Water Insight B.V. (Collaboration)
- National Research Council (Collaboration)
- Meteorological Office UK (Collaboration)
- CENTRE FOR ENVIRONMENT, FISHERIES AND AQUACULTURE SCIENCE (Collaboration)
- Norwegian Institute for Water Research (NIVA) (Collaboration)
- Free University of Amsterdam (Collaboration)
- Dow Chemical Company (DOW) (Collaboration)
- European Space Agency (Collaboration)
- Italian National Institute of Oceanography and Experimental Geophysics (OGS) (Collaboration)
- Middle East Technical University (Collaboration)
- European Commission (Collaboration)
Publications
M Grant
(2014)
Sharing experiences with big data web services infrastructure to support the Data-CORE
in GEO European Projects Workshop
| Description | The grant provided infrastructure that could support modern and novel "big data" data services in a robust fashion, with the primary purposes of allowing people without this infrastructure to supply data, to disseminate information on how to set up such infrastructure and to trailblaze new methods to explore whether they are worthwhile for the Earth observation, modelling and oceanography communities. |
| Exploitation Route | 1. Deployment of their data on this cluster 2. Development of their own cluster following some of the instructions produced by this grant 3. Adoption (or justified rejection!) of new methods, based on exploratory work done with this infrastructure 4. Further development of methods, based on experiences with handling extremely large datasets with this infrastructure |
| Sectors | Aerospace, Defence and Marine,Digital/Communication/Information Technologies (including Software),Environment |
| URL | https://vortices.npm.ac.uk/ |
| Description | The data services have been used to supply data for a range of international projects, including FP7, ESA and UNEP funded ones, and a number of internal and external collaborators. We hope that a good experience with robust data services will lead to greater deployment of similar systems, which are essential to deal with modern data volumes and usage patterns. The data served has been used globally by a range of research communities (mainly Earth Observation, modelling and oceanography), but are also publically accessible to all. Exploration of the new methods has resulted in requests for advice from numerous centres across Europe. We have also organised a workshop on big data services, where our findings were disseminated and discussed with other interested parties (mainly ICT oriented) and have shared our key experiences with the GEO European Projects Workshop. |
| First Year Of Impact | 2014 |
| Sector | Aerospace, Defence and Marine,Digital/Communication/Information Technologies (including Software),Environment |
| Impact Types | Policy & public services |
| Description | ESA Climate Change Initiative |
| Organisation | European Space Agency |
| Country | France |
| Sector | Public |
| PI Contribution | Provision of data services for the Ocean Colour Climate Change Initiative project. Includes novel use of WCPS querying to exploit the availability of per pixel uncertainty estimates and water type classification embedded in the data. |
| Collaborator Contribution | 15 year unified time series of SeaWiFS, MODIS and MERIS OC data. |
| Impact | Normally these data are available only over FTP. For the size of this dataset, subsetting services like OPeNDAP are essential and were explicitly requested by end users, ranging from ecosystem modellers, Earth observation specialists to consultancies, from Japan to Europe to America. Primary data at https://rsg.pml.ac.uk/thredds/catalog-cci.html , portal at http://www.oceancolour.org/ |
| Start Year | 2014 |
| Description | Ecosystem modellers collaboration |
| Organisation | Plymouth Marine Laboratory |
| Country | United Kingdom |
| Sector | Academic/University |
| PI Contribution | Internal collaboration with the modelling group within PML, which wanted to use their model outputs via OPeNDAP data services, but lacked the infrastructure and experience to establish them. |
| Collaborator Contribution | Data are publically available and potentially to be used by other collaborators working on the open source FVCOM model. |
| Impact | Additional freely available test data from a contemporary model. |
| Start Year | 2014 |
| Description | FP7 Aqua-Users |
| Organisation | Dow Chemical Company (DOW) |
| Department | Dow Haltermann |
| Country | United Kingdom |
| Sector | Private |
| PI Contribution | Providing the user interface and underlying data services |
| Collaborator Contribution | Data supply (freely available) |
| Impact | From the big data cluster standpoint, the main output is the additional data availability and ongoing development work (plus sharing of experiences). |
| Start Year | 2014 |
| Description | FP7 Aqua-Users |
| Organisation | Dow Chemical Company (DOW) |
| Department | Dow Haltermann |
| Country | United Kingdom |
| Sector | Private |
| PI Contribution | Providing the user interface and underlying data services |
| Collaborator Contribution | Data supply (freely available) |
| Impact | From the big data cluster standpoint, the main output is the additional data availability and ongoing development work (plus sharing of experiences). |
| Start Year | 2014 |
| Description | FP7 Aqua-Users |
| Organisation | European Commission |
| Country | European Union (EU) |
| Sector | Public |
| PI Contribution | Providing the user interface and underlying data services |
| Collaborator Contribution | Data supply (freely available) |
| Impact | From the big data cluster standpoint, the main output is the additional data availability and ongoing development work (plus sharing of experiences). |
| Start Year | 2014 |
| Description | FP7 Aqua-Users |
| Organisation | Free University of Amsterdam |
| Department | Department of Ecological Science |
| Country | Netherlands |
| Sector | Academic/University |
| PI Contribution | Providing the user interface and underlying data services |
| Collaborator Contribution | Data supply (freely available) |
| Impact | From the big data cluster standpoint, the main output is the additional data availability and ongoing development work (plus sharing of experiences). |
| Start Year | 2014 |
| Description | FP7 Aqua-Users |
| Organisation | Norwegian Institute for Water Research (NIVA) |
| Country | Norway |
| Sector | Public |
| PI Contribution | Providing the user interface and underlying data services |
| Collaborator Contribution | Data supply (freely available) |
| Impact | From the big data cluster standpoint, the main output is the additional data availability and ongoing development work (plus sharing of experiences). |
| Start Year | 2014 |
| Description | FP7 Aqua-Users |
| Organisation | University of Lisbon |
| Department | Faculty of Science (FCUL) |
| Country | Portugal |
| Sector | Academic/University |
| PI Contribution | Providing the user interface and underlying data services |
| Collaborator Contribution | Data supply (freely available) |
| Impact | From the big data cluster standpoint, the main output is the additional data availability and ongoing development work (plus sharing of experiences). |
| Start Year | 2014 |
| Description | FP7 Aqua-Users |
| Organisation | Water Insight B.V. |
| Country | Netherlands |
| Sector | Private |
| PI Contribution | Providing the user interface and underlying data services |
| Collaborator Contribution | Data supply (freely available) |
| Impact | From the big data cluster standpoint, the main output is the additional data availability and ongoing development work (plus sharing of experiences). |
| Start Year | 2014 |
| Description | FP7 Earth2Observe |
| Organisation | Deltares |
| Country | Netherlands |
| Sector | Private |
| PI Contribution | These subset of partners needed to supply large volumes of data over WxS data services and lacked the capability. PML is hosting their data and assisting them to set up their own services, when desired. |
| Collaborator Contribution | This project will produce a major reevaluation of the global water cycle; all of these data will be freely available. |
| Impact | So far, a number of precursor datasets have been published; the reanalysis datasets are due to be published in the next year or two. The collaboration includes ICT specialists, modellers and earth observation scientists. |
| Start Year | 2013 |
| Description | FP7 Earth2Observe |
| Organisation | European Centre for Medium Range Weather Forecasting ECMWF |
| Country | United Kingdom |
| Sector | Public |
| PI Contribution | These subset of partners needed to supply large volumes of data over WxS data services and lacked the capability. PML is hosting their data and assisting them to set up their own services, when desired. |
| Collaborator Contribution | This project will produce a major reevaluation of the global water cycle; all of these data will be freely available. |
| Impact | So far, a number of precursor datasets have been published; the reanalysis datasets are due to be published in the next year or two. The collaboration includes ICT specialists, modellers and earth observation scientists. |
| Start Year | 2013 |
| Description | FP7 Earth2Observe |
| Organisation | European Commission |
| Country | European Union (EU) |
| Sector | Public |
| PI Contribution | These subset of partners needed to supply large volumes of data over WxS data services and lacked the capability. PML is hosting their data and assisting them to set up their own services, when desired. |
| Collaborator Contribution | This project will produce a major reevaluation of the global water cycle; all of these data will be freely available. |
| Impact | So far, a number of precursor datasets have been published; the reanalysis datasets are due to be published in the next year or two. The collaboration includes ICT specialists, modellers and earth observation scientists. |
| Start Year | 2013 |
| Description | FP7 Earth2Observe |
| Organisation | Free University of Amsterdam |
| Country | Netherlands |
| Sector | Academic/University |
| PI Contribution | These subset of partners needed to supply large volumes of data over WxS data services and lacked the capability. PML is hosting their data and assisting them to set up their own services, when desired. |
| Collaborator Contribution | This project will produce a major reevaluation of the global water cycle; all of these data will be freely available. |
| Impact | So far, a number of precursor datasets have been published; the reanalysis datasets are due to be published in the next year or two. The collaboration includes ICT specialists, modellers and earth observation scientists. |
| Start Year | 2013 |
| Description | FP7 Earth2Observe |
| Organisation | National Research Council |
| Country | Italy |
| Sector | Public |
| PI Contribution | These subset of partners needed to supply large volumes of data over WxS data services and lacked the capability. PML is hosting their data and assisting them to set up their own services, when desired. |
| Collaborator Contribution | This project will produce a major reevaluation of the global water cycle; all of these data will be freely available. |
| Impact | So far, a number of precursor datasets have been published; the reanalysis datasets are due to be published in the next year or two. The collaboration includes ICT specialists, modellers and earth observation scientists. |
| Start Year | 2013 |
| Description | FP7 Earth2Observe |
| Organisation | Vienna University of Technology |
| Country | Austria |
| Sector | Academic/University |
| PI Contribution | These subset of partners needed to supply large volumes of data over WxS data services and lacked the capability. PML is hosting their data and assisting them to set up their own services, when desired. |
| Collaborator Contribution | This project will produce a major reevaluation of the global water cycle; all of these data will be freely available. |
| Impact | So far, a number of precursor datasets have been published; the reanalysis datasets are due to be published in the next year or two. The collaboration includes ICT specialists, modellers and earth observation scientists. |
| Start Year | 2013 |
| Description | FP7 OPEC |
| Organisation | Centre For Environment, Fisheries And Aquaculture Science |
| Country | United Kingdom |
| Sector | Public |
| PI Contribution | These partners in FP7 OPEC have specifically made use of the data services provided by the big data cluster |
| Collaborator Contribution | The data they have contributed are now freely available. |
| Impact | Major outputs are the contribution towards the OPEC project, which aims to develop and evaluate ecosystem forecast tools to help assess and manage the risks posed by human activities on the marine environment. The specific outputs from this collaboration are much faster and more reliable data services for the project and its users [primarily ecosystem modellers and earth observation specialists], and the greater availability of the datasets produced by the project [general public and any other interested users]. The data services support a portal - http://marine-opec.eu/portal.html |
| Start Year | 2014 |
| Description | FP7 OPEC |
| Organisation | Danish Meteorological Institute (DMI) |
| Country | Denmark |
| Sector | Public |
| PI Contribution | These partners in FP7 OPEC have specifically made use of the data services provided by the big data cluster |
| Collaborator Contribution | The data they have contributed are now freely available. |
| Impact | Major outputs are the contribution towards the OPEC project, which aims to develop and evaluate ecosystem forecast tools to help assess and manage the risks posed by human activities on the marine environment. The specific outputs from this collaboration are much faster and more reliable data services for the project and its users [primarily ecosystem modellers and earth observation specialists], and the greater availability of the datasets produced by the project [general public and any other interested users]. The data services support a portal - http://marine-opec.eu/portal.html |
| Start Year | 2014 |
| Description | FP7 OPEC |
| Organisation | European Commission |
| Country | European Union (EU) |
| Sector | Public |
| PI Contribution | These partners in FP7 OPEC have specifically made use of the data services provided by the big data cluster |
| Collaborator Contribution | The data they have contributed are now freely available. |
| Impact | Major outputs are the contribution towards the OPEC project, which aims to develop and evaluate ecosystem forecast tools to help assess and manage the risks posed by human activities on the marine environment. The specific outputs from this collaboration are much faster and more reliable data services for the project and its users [primarily ecosystem modellers and earth observation specialists], and the greater availability of the datasets produced by the project [general public and any other interested users]. The data services support a portal - http://marine-opec.eu/portal.html |
| Start Year | 2014 |
| Description | FP7 OPEC |
| Organisation | Italian National Institute of Oceanography and Experimental Geophysics (OGS) |
| Country | Italy |
| Sector | Public |
| PI Contribution | These partners in FP7 OPEC have specifically made use of the data services provided by the big data cluster |
| Collaborator Contribution | The data they have contributed are now freely available. |
| Impact | Major outputs are the contribution towards the OPEC project, which aims to develop and evaluate ecosystem forecast tools to help assess and manage the risks posed by human activities on the marine environment. The specific outputs from this collaboration are much faster and more reliable data services for the project and its users [primarily ecosystem modellers and earth observation specialists], and the greater availability of the datasets produced by the project [general public and any other interested users]. The data services support a portal - http://marine-opec.eu/portal.html |
| Start Year | 2014 |
| Description | FP7 OPEC |
| Organisation | Middle East Technical University |
| Department | Institute of Marine Sciences |
| Country | Turkey |
| Sector | Academic/University |
| PI Contribution | These partners in FP7 OPEC have specifically made use of the data services provided by the big data cluster |
| Collaborator Contribution | The data they have contributed are now freely available. |
| Impact | Major outputs are the contribution towards the OPEC project, which aims to develop and evaluate ecosystem forecast tools to help assess and manage the risks posed by human activities on the marine environment. The specific outputs from this collaboration are much faster and more reliable data services for the project and its users [primarily ecosystem modellers and earth observation specialists], and the greater availability of the datasets produced by the project [general public and any other interested users]. The data services support a portal - http://marine-opec.eu/portal.html |
| Start Year | 2014 |
| Description | FP7 OPEC |
| Organisation | Plymouth Marine Laboratory |
| Country | United Kingdom |
| Sector | Academic/University |
| PI Contribution | These partners in FP7 OPEC have specifically made use of the data services provided by the big data cluster |
| Collaborator Contribution | The data they have contributed are now freely available. |
| Impact | Major outputs are the contribution towards the OPEC project, which aims to develop and evaluate ecosystem forecast tools to help assess and manage the risks posed by human activities on the marine environment. The specific outputs from this collaboration are much faster and more reliable data services for the project and its users [primarily ecosystem modellers and earth observation specialists], and the greater availability of the datasets produced by the project [general public and any other interested users]. The data services support a portal - http://marine-opec.eu/portal.html |
| Start Year | 2014 |
| Description | FP7 OPEC |
| Organisation | Technical University of Denmark |
| Country | Denmark |
| Sector | Academic/University |
| PI Contribution | These partners in FP7 OPEC have specifically made use of the data services provided by the big data cluster |
| Collaborator Contribution | The data they have contributed are now freely available. |
| Impact | Major outputs are the contribution towards the OPEC project, which aims to develop and evaluate ecosystem forecast tools to help assess and manage the risks posed by human activities on the marine environment. The specific outputs from this collaboration are much faster and more reliable data services for the project and its users [primarily ecosystem modellers and earth observation specialists], and the greater availability of the datasets produced by the project [general public and any other interested users]. The data services support a portal - http://marine-opec.eu/portal.html |
| Start Year | 2014 |
| Description | NCOF |
| Organisation | Meteorological Office UK |
| Country | United Kingdom |
| Sector | Academic/University |
| PI Contribution | Provision of data services; the Met Office was prevented from doing so due to on-site security restrictions |
| Collaborator Contribution | Public availability of the MRCS model outputs for the UK area, then superseded by FOAM. |
| Impact | Public availability of the MRCS model outputs for the UK area, then superseded by FOAM. Data feed at http://ncof.pml.ac.uk/ |
| Start Year | 2008 |
| Description | NCOF |
| Organisation | The National Centre for Ocean Forecasting |
| Country | United Kingdom |
| Sector | Private |
| PI Contribution | Provision of data services; the Met Office was prevented from doing so due to on-site security restrictions |
| Collaborator Contribution | Public availability of the MRCS model outputs for the UK area, then superseded by FOAM. |
| Impact | Public availability of the MRCS model outputs for the UK area, then superseded by FOAM. Data feed at http://ncof.pml.ac.uk/ |
| Start Year | 2008 |
| Description | NEODAAS |
| Organisation | Plymouth Marine Laboratory |
| Country | United Kingdom |
| Sector | Academic/University |
| PI Contribution | Experimental data services for specific areas of interest. NEODAAS has had a long term interest in web services, but never fully deployed them. This test case may help tip the balance. |
| Collaborator Contribution | Availability of a ~20 year satellite time series by new access methods, albeit only in small parts. The intention is to make the full series available, if technically possible. |
| Impact | Availability of a ~20 year satellite time series by new access methods, albeit only in small parts. The intention is to make the full series available, if technically possible. Collaboration between computing specialists and earth observation scientists. |
| Start Year | 2006 |
| Description | Transboundary Waters Assessment Programme data services |
| Organisation | Plymouth Marine Laboratory |
| Country | United Kingdom |
| Sector | Academic/University |
| PI Contribution | The project wanted to add its data outputs to some websites and portals, which required WxS data services. The team did not have the capability or infrastructure themselves. |
| Collaborator Contribution | Additional public data. |
| Impact | Additional public data, e.g. http://twapgeoportal.grid.unep.ch/geonetwork/srv/eng/metadata.show?id=1141&currTab=simple and https://rsg.pml.ac.uk/thredds/catalog-twap.html Collaboration includes oceanographers / remote sensing scientists and computing. |
| Start Year | 2014 |
| Description | Transboundary Waters Assessment Programme data services |
| Organisation | United Nations (UN) |
| Department | United Nations Environment Programme |
| Country | Kenya |
| Sector | Charity/Non Profit |
| PI Contribution | The project wanted to add its data outputs to some websites and portals, which required WxS data services. The team did not have the capability or infrastructure themselves. |
| Collaborator Contribution | Additional public data. |
| Impact | Additional public data, e.g. http://twapgeoportal.grid.unep.ch/geonetwork/srv/eng/metadata.show?id=1141&currTab=simple and https://rsg.pml.ac.uk/thredds/catalog-twap.html Collaboration includes oceanographers / remote sensing scientists and computing. |
| Start Year | 2014 |
| Description | Western Channel Observatory |
| Organisation | Plymouth Marine Laboratory |
| Country | United Kingdom |
| Sector | Academic/University |
| PI Contribution | Provision of WxS data services and integration into portals for WCO live outputs, which the WCO group did not have the capability to do themselves. |
| Collaborator Contribution | Greater availability of near-real-time insitu data from the WC Observatory |
| Impact | A number of data streams are potentially available. The most recent is from the new Penlee on-shore observation station - https://rsg.pml.ac.uk/thredds/catalog-penlee.html |
| Start Year | 2008 |
| Title | Big data cluster |
| Description | The purpose of this grant was to fund infrastructure to provide reliable and robust data and processing services, and particularly to explore the technology in the context of very large datasets, with the aim of improving the technology, the variety and number of deployments and the knowledge in the community about these services / how to set them up. |
| Type Of Technology | New/Improved Technique/Technology |
| Year Produced | 2014 |
| Impact | Significant numbers of large datasets made available to an actively global userbase. Dissemination of recommendations and key setup information to a range of organisations, from ICT specialists to scientific organisations wanting similar systems but lacking the experience. Exploration of cutting-edge technologies for handling extremely large datasets, both on a service level and as a user experience, plus dissemination of these experiences. |
| URL | https://vortices.npm.ac.uk/ |