EO Data Hub
Lead Research Organisation:
University of Leicester
Department Name: National Centre for Earth Observation
Abstract
Earth observation data can provide a wealth of information by virtue of its comprehensive geo-temporal coverage, the range of instruments and sensing platforms and the sophistication with which measurements can be derived from the Earth system. These can be applied to obtain insights into the interaction between human activity and the Earth system.
Particularly important factors in recent times have been the recognition of the value of both public service (state-funded through space agencies) and commercial constellations, good access to key data sets from trusted sources and the desire to combine data sets for high level product or information services relevant to environmental sciences. Hitherto, the UK strategy has been to invest in data centres, hubs and platforms via Europe and both science assets such as the Centre for Environmental Data Analysis (CEDA) and UK software and data platform companies have been successful in obtaining funding in a European context. This project is a very important first step driven by the UK's new National Space Strategy (NSS) which identifies the opportunity of market growth in EO, the need for world-leading science and technology e.g., the fight against climate change, and the value of international partnerships.
The sheer scale of EO datasets can presents barriers to its effective exploitation due to a significant growth in types of sensors, in data volumes, and in data products which are essential outputs from satellites for science to services. Furthermore, practitioners often wish to combine EO with sensor networks, model data and, in a new challenge, digital twins of the Earth.
The EO DataHub seeks to meet these challenges, capitalise on UK strengths and take the opportunity to create a first UK infrastructure capable of serving public and commercial EO data, alongside other data sources, to a wide range of users from science to industry to government. The key attributes of this pre-operational service, which is intended to be free in this phase to UK users at the point of use, are the following:
- A federated approach using UK-specific software infrastructure which is state-of-the-art, and demonstrably keeps the UK at the forefront of digital EO technologies, reducing data downloads and data volumes for storage
- Platforms for specific sectors which are judged to enable the most benefit in this phase of the project; one will be climate.
- Pilot projects and champion users which will support progression of the concept, specifications, evaluation and demonstration.
Software wrappers to allow algorithms to run at data sources (selected sources)
- Access to scientific, public and commercial data sets; the last by procurement and negotiation.
- A central place for NERC community EO data sets, alongside its Environmental Data Service, specifically serving the outputs of the UKEO Climate Information Service (UKEO CIS) and model data from the Coupled Model Intercomparison Project, CMIP6.
Through the provision of a shared software infrastructure and user services, the data hub will seek to address the issues around fragmentation of data access and services experienced by the user community. Data streams from the academic and commercial sectors will build on and provide continuity from previous UK investments, including high quality, Essential Climate Variable UK-produced datasets and UK-specific datasets; high-profile climate model simulations - CMIP and CORDEX; access to operational data streams and commercial data streams. By providing an innovation laboratory, the EO Data Hub expects to support academic, industry and government users leading to 1000+ users; five large-scale, impactful activities with business or national stakeholders; support to at least 10 SMEs per year and 10 government users.
Particularly important factors in recent times have been the recognition of the value of both public service (state-funded through space agencies) and commercial constellations, good access to key data sets from trusted sources and the desire to combine data sets for high level product or information services relevant to environmental sciences. Hitherto, the UK strategy has been to invest in data centres, hubs and platforms via Europe and both science assets such as the Centre for Environmental Data Analysis (CEDA) and UK software and data platform companies have been successful in obtaining funding in a European context. This project is a very important first step driven by the UK's new National Space Strategy (NSS) which identifies the opportunity of market growth in EO, the need for world-leading science and technology e.g., the fight against climate change, and the value of international partnerships.
The sheer scale of EO datasets can presents barriers to its effective exploitation due to a significant growth in types of sensors, in data volumes, and in data products which are essential outputs from satellites for science to services. Furthermore, practitioners often wish to combine EO with sensor networks, model data and, in a new challenge, digital twins of the Earth.
The EO DataHub seeks to meet these challenges, capitalise on UK strengths and take the opportunity to create a first UK infrastructure capable of serving public and commercial EO data, alongside other data sources, to a wide range of users from science to industry to government. The key attributes of this pre-operational service, which is intended to be free in this phase to UK users at the point of use, are the following:
- A federated approach using UK-specific software infrastructure which is state-of-the-art, and demonstrably keeps the UK at the forefront of digital EO technologies, reducing data downloads and data volumes for storage
- Platforms for specific sectors which are judged to enable the most benefit in this phase of the project; one will be climate.
- Pilot projects and champion users which will support progression of the concept, specifications, evaluation and demonstration.
Software wrappers to allow algorithms to run at data sources (selected sources)
- Access to scientific, public and commercial data sets; the last by procurement and negotiation.
- A central place for NERC community EO data sets, alongside its Environmental Data Service, specifically serving the outputs of the UKEO Climate Information Service (UKEO CIS) and model data from the Coupled Model Intercomparison Project, CMIP6.
Through the provision of a shared software infrastructure and user services, the data hub will seek to address the issues around fragmentation of data access and services experienced by the user community. Data streams from the academic and commercial sectors will build on and provide continuity from previous UK investments, including high quality, Essential Climate Variable UK-produced datasets and UK-specific datasets; high-profile climate model simulations - CMIP and CORDEX; access to operational data streams and commercial data streams. By providing an innovation laboratory, the EO Data Hub expects to support academic, industry and government users leading to 1000+ users; five large-scale, impactful activities with business or national stakeholders; support to at least 10 SMEs per year and 10 government users.
Organisations
Description | Collaboration with EOCIS project to facilitate processing of data, archiving of data outputs and development of software tools and services |
Organisation | University of Reading |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | The EO DataHub (EODH) project has procured storage for the JASMIN data analysis system (jasmin.ac.uk) to facilitate storage and processing of data outputs from the EOCIS project. The EODH is providing data archiving services via NERC's designated data centre for Earth Observation data - CEDA (www.ceda.ac.uk) and has developed software services to enable data from EOCIS to be discovered (search tools) and accessed. |
Collaborator Contribution | The EOCIS project team have provided expert information in the nature of the data to be produced to support it's processing, archiving and dissemination to downstream user communities |
Impact | Software has been developed to facilitate data access services. Computer hardware has been procured to enable data production and storage. The collaboration covers Earth observation and climate sectors. |
Start Year | 2023 |
Description | Software development of the Hub Platform |
Organisation | Telespazio Vega (IDEAS) |
Country | United Kingdom |
Sector | Private |
PI Contribution | The team has developed the invitation to tender to specify the work to be carried out and procured. Telespazio were the successful supplier for this tender. |
Collaborator Contribution | Telespazio have been awarded a contract from NERC to develop a major software component for the EO DataHub, the "Hub Platform". This software is under development. It will facilitate users of the EO DataHub to more readily access Earth observation data and create software applications which use the data to obtain information, provide insights and analysis for the benefit of industry, public sector and academic community. |
Impact | Software currently under development. Multi-disciplinary: Earth observation, climate, Geographic Information Systems (GIS) |
Start Year | 2023 |
Description | User Pilot: Open Data Cube pilot for South of England |
Organisation | Joint Nature Conservation Committee |
Country | United Kingdom |
Sector | Private |
PI Contribution | The team developed an invitation to tender to elicit proposals from industry, academia and public sector for software applications suitable for exploiting the EO DataHub in the future. The JNCC were one of the successful suppliers for this tender. |
Collaborator Contribution | JNCC ran a pilot project to demonstrate an open data cube of satellite data for the analysis of land use for a UK National Park. |
Impact | A demonstrator application, report and presentation of findings. Investigation of the Open Data Cube software. |
Start Year | 2023 |
Description | User Pilot: Robust shoreline and intertidal mapping |
Organisation | National Oceanography Centre |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | The team developed an invitation to tender to elicit proposals from industry, academia and public sector for software applications suitable for exploiting the EO DataHub in the future. NOC and Argans worked in a consortium and were successful suppliers for this tender. |
Collaborator Contribution | The partners ran a pilot project to investigate the development of a software application to provide accurate mapping of shorelines using satellite imagery. |
Impact | A demonstrator application, report and presentation of findings |
Start Year | 2023 |
Description | User Pilot: Seagrass mapping |
Organisation | CGI |
Country | Canada |
Sector | Private |
PI Contribution | The team developed an invitation to tender to elicit proposals from industry, academia and public sector for software applications suitable for exploiting the EO DataHub in the future. CGI UK were one of the successful suppliers for this tender. |
Collaborator Contribution | CGI ran a pilot project to demonstrate an application which maps seagrass coverage in coastal areas using satellite imagery |
Impact | A demonstrator application, report and presentation of findings |
Start Year | 2023 |
Description | User Pilot: Ship routing, including ice-covered waters |
Organisation | British Antarctic Survey |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | The team developed an invitation to tender to elicit proposals from industry, academia and public sector for software applications suitable for exploiting the EO DataHub in the future. BAS were one of the successful suppliers for this tender. |
Collaborator Contribution | BAS ran a pilot project to demonstrate the application of AI technology and satellite imagery to assist with ship navigation through ice flows in Antarctic waters. |
Impact | A demonstrator application, report and presentation of findings |
Start Year | 2023 |
Description | User Pilot: Ship routing, including ice-covered waters |
Organisation | British Antarctic Survey |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | The team developed an invitation to tender to elicit proposals from industry, academia and public sector for software applications suitable for exploiting the EO DataHub in the future. The British Antarctic Survey were one of the successful suppliers for this tender. |
Collaborator Contribution | The British Antarctic Survey ran a pilot project to demonstrate an application which uses AI and satellite imagery to help ships navigate through sea ice. |
Impact | A demonstrator application, report and presentation of findings |
Start Year | 2023 |
Title | STAC Generator |
Description | The STAC Generator provides the framework and access to shared tools. The framework allows you to build generators to get metadata from file objects using plugins to change the source of the files, the output of the metadata and the processing chain which extracts the metadata. The framework leverages a modular, plugin architecture to allow users to modify the workflow to fit their needs. |
Type Of Technology | Software |
Year Produced | 2024 |
Open Source License? | Yes |
Impact | Make part of the CEDA Archive accessible to externals using STAC and in particular to EODH |
Title | STAC Notebooks |
Description | A set of Notebooks to explore the STAC API developed in the EODH |
Type Of Technology | Software |
Year Produced | 2024 |
Open Source License? | Yes |
Impact | Using the notebooks is lowering the barrier of access of the APIs and give concrete examples of how to use the software supporting the users with up-to-date documentation and recipes |
Title | kerchunk-builder |
Description | A pipeline to convert NEtCDF file into Kerchunk data format. Kerchunk is a library that provides a unified way to represent a variety of chunked, compressed data formats (e.g. NetCDF/HDF5, GRIB2, TIFF, ), allowing efficient access to the data from traditional file systems or cloud object storage. Documentation available here: https://cedadev.github.io/kerchunk-builder/ |
Type Of Technology | Software |
Year Produced | 2024 |
Impact | The pipeline has been developed under the framework of EODH, and it has been used over a numerous of projects already (as such CCI) |
Description | EO DataHub Project Outline for Industry Day, 13 March 2023 |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Industry/Business |
Results and Impact | Presentation of the EO DataHub project to representatives from industry in order to engage them as suppliers for subsequent tenders for contracts to develop software and data services. |
Year(s) Of Engagement Activity | 2023 |
Description | Presentation "EO DataHub Key Features" to the EO DataHub Users and Stakeholders Forum, 27 July 2023 |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Industry/Business |
Results and Impact | A presentation on the EO DataHub project to a User and Stakeholder forum organised to disseminate the work of the project and engage with individuals and groups from the UK EO sector interested in being suppliers to its programme of work, or being users of the resulting software systems and data it provides. |
Year(s) Of Engagement Activity | 2023 |
Description | Presentation on the EO DataHub project at the NCEO Conference 2023, 11-13 September, Leeds |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Other audiences |
Results and Impact | Present to the NCEO community the EO DataHub project describing its purpose and scope and how they can engage with its activities. |
Year(s) Of Engagement Activity | 2023 |
Description | Presentation on the EO DataHub project to the Earth Observation Centre of Excellence group, 26 June 2023 |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | Presentation on the EO DataHub project to the EO Centre of Excellence a working group which brings together public sector users of Earth observation data. The purpose of the presentation was to raise awareness of the EO DataHub and the potential use and benefits to users of EO data in the public sector |
Year(s) Of Engagement Activity | 2023 |
Description | Presentation on the EO DataHub project to the NCEO Staff Conference, London, 9 January 2024 |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Other audiences |
Results and Impact | Provide an overview of the EO DataHub project to the NCEO community and explain to them how they can engage with this activity particular with respect to data outputs from the related EOCIS project. |
Year(s) Of Engagement Activity | 2024 |