EO Data Hub

Lead Research Organisation: University of Leicester
Department Name: National Centre for Earth Observation

Abstract

Earth observation data can provide a wealth of information by virtue of its comprehensive geo-temporal coverage, the range of instruments and sensing platforms and the sophistication with which measurements can be derived from the Earth system. These can be applied to obtain insights into the interaction between human activity and the Earth system.

Particularly important factors in recent times have been the recognition of the value of both public service (state-funded through space agencies) and commercial constellations, good access to key data sets from trusted sources and the desire to combine data sets for high level product or information services relevant to environmental sciences. Hitherto, the UK strategy has been to invest in data centres, hubs and platforms via Europe and both science assets such as the Centre for Environmental Data Analysis (CEDA) and UK software and data platform companies have been successful in obtaining funding in a European context. This project is a very important first step driven by the UK's new National Space Strategy (NSS) which identifies the opportunity of market growth in EO, the need for world-leading science and technology e.g., the fight against climate change, and the value of international partnerships.

The sheer scale of EO datasets can presents barriers to its effective exploitation due to a significant growth in types of sensors, in data volumes, and in data products which are essential outputs from satellites for science to services. Furthermore, practitioners often wish to combine EO with sensor networks, model data and, in a new challenge, digital twins of the Earth.

The EO DataHub seeks to meet these challenges, capitalise on UK strengths and take the opportunity to create a first UK infrastructure capable of serving public and commercial EO data, alongside other data sources, to a wide range of users from science to industry to government. The key attributes of this pre-operational service, which is intended to be free in this phase to UK users at the point of use, are the following:
- A federated approach using UK-specific software infrastructure which is state-of-the-art, and demonstrably keeps the UK at the forefront of digital EO technologies, reducing data downloads and data volumes for storage
- Platforms for specific sectors which are judged to enable the most benefit in this phase of the project; one will be climate.
- Pilot projects and champion users which will support progression of the concept, specifications, evaluation and demonstration.
Software wrappers to allow algorithms to run at data sources (selected sources)
- Access to scientific, public and commercial data sets; the last by procurement and negotiation.
- A central place for NERC community EO data sets, alongside its Environmental Data Service, specifically serving the outputs of the UKEO Climate Information Service (UKEO CIS) and model data from the Coupled Model Intercomparison Project, CMIP6.

Through the provision of a shared software infrastructure and user services, the data hub will seek to address the issues around fragmentation of data access and services experienced by the user community. Data streams from the academic and commercial sectors will build on and provide continuity from previous UK investments, including high quality, Essential Climate Variable UK-produced datasets and UK-specific datasets; high-profile climate model simulations - CMIP and CORDEX; access to operational data streams and commercial data streams. By providing an innovation laboratory, the EO Data Hub expects to support academic, industry and government users leading to 1000+ users; five large-scale, impactful activities with business or national stakeholders; support to at least 10 SMEs per year and 10 government users.

Publications

10 25 50
 
Description Collaboration with EOCIS project to facilitate processing of data, archiving of data outputs and development of software tools and services 
Organisation University of Reading
Country United Kingdom 
Sector Academic/University 
PI Contribution The EO DataHub (EODH) project has procured storage for the JASMIN data analysis system (jasmin.ac.uk) to facilitate storage and processing of data outputs from the EOCIS project. The EODH is providing data archiving services via NERC's designated data centre for Earth Observation data - CEDA (www.ceda.ac.uk) and has developed software services to enable data from EOCIS to be discovered (search tools) and accessed.
Collaborator Contribution The EOCIS project team have provided expert information in the nature of the data to be produced to support it's processing, archiving and dissemination to downstream user communities
Impact Software has been developed to facilitate data access services. Computer hardware has been procured to enable data production and storage. The collaboration covers Earth observation and climate sectors.
Start Year 2023
 
Description Software development of the Hub Platform 
Organisation Telespazio Vega (IDEAS)
Country United Kingdom 
Sector Private 
PI Contribution The team has developed the invitation to tender to specify the work to be carried out and procured. Telespazio were the successful supplier for this tender.
Collaborator Contribution Telespazio have been awarded a contract from NERC to develop a major software component for the EO DataHub, the "Hub Platform". This software is under development. It will facilitate users of the EO DataHub to more readily access Earth observation data and create software applications which use the data to obtain information, provide insights and analysis for the benefit of industry, public sector and academic community.
Impact Software currently under development. Multi-disciplinary: Earth observation, climate, Geographic Information Systems (GIS)
Start Year 2023
 
Description User Pilot: Open Data Cube pilot for South of England 
Organisation Joint Nature Conservation Committee
Country United Kingdom 
Sector Private 
PI Contribution The team developed an invitation to tender to elicit proposals from industry, academia and public sector for software applications suitable for exploiting the EO DataHub in the future. The JNCC were one of the successful suppliers for this tender.
Collaborator Contribution JNCC ran a pilot project to demonstrate an open data cube of satellite data for the analysis of land use for a UK National Park.
Impact A demonstrator application, report and presentation of findings. Investigation of the Open Data Cube software.
Start Year 2023
 
Description User Pilot: Robust shoreline and intertidal mapping 
Organisation National Oceanography Centre
Country United Kingdom 
Sector Academic/University 
PI Contribution The team developed an invitation to tender to elicit proposals from industry, academia and public sector for software applications suitable for exploiting the EO DataHub in the future. NOC and Argans worked in a consortium and were successful suppliers for this tender.
Collaborator Contribution The partners ran a pilot project to investigate the development of a software application to provide accurate mapping of shorelines using satellite imagery.
Impact A demonstrator application, report and presentation of findings
Start Year 2023
 
Description User Pilot: Seagrass mapping 
Organisation CGI
Country Canada 
Sector Private 
PI Contribution The team developed an invitation to tender to elicit proposals from industry, academia and public sector for software applications suitable for exploiting the EO DataHub in the future. CGI UK were one of the successful suppliers for this tender.
Collaborator Contribution CGI ran a pilot project to demonstrate an application which maps seagrass coverage in coastal areas using satellite imagery
Impact A demonstrator application, report and presentation of findings
Start Year 2023
 
Description User Pilot: Ship routing, including ice-covered waters 
Organisation British Antarctic Survey
Country United Kingdom 
Sector Academic/University 
PI Contribution The team developed an invitation to tender to elicit proposals from industry, academia and public sector for software applications suitable for exploiting the EO DataHub in the future. BAS were one of the successful suppliers for this tender.
Collaborator Contribution BAS ran a pilot project to demonstrate the application of AI technology and satellite imagery to assist with ship navigation through ice flows in Antarctic waters.
Impact A demonstrator application, report and presentation of findings
Start Year 2023
 
Description User Pilot: Ship routing, including ice-covered waters 
Organisation British Antarctic Survey
Country United Kingdom 
Sector Academic/University 
PI Contribution The team developed an invitation to tender to elicit proposals from industry, academia and public sector for software applications suitable for exploiting the EO DataHub in the future. The British Antarctic Survey were one of the successful suppliers for this tender.
Collaborator Contribution The British Antarctic Survey ran a pilot project to demonstrate an application which uses AI and satellite imagery to help ships navigate through sea ice.
Impact A demonstrator application, report and presentation of findings
Start Year 2023
 
Title STAC Generator 
Description The STAC Generator provides the framework and access to shared tools. The framework allows you to build generators to get metadata from file objects using plugins to change the source of the files, the output of the metadata and the processing chain which extracts the metadata. The framework leverages a modular, plugin architecture to allow users to modify the workflow to fit their needs. 
Type Of Technology Software 
Year Produced 2024 
Open Source License? Yes  
Impact Make part of the CEDA Archive accessible to externals using STAC and in particular to EODH 
 
Title STAC Notebooks 
Description A set of Notebooks to explore the STAC API developed in the EODH 
Type Of Technology Software 
Year Produced 2024 
Open Source License? Yes  
Impact Using the notebooks is lowering the barrier of access of the APIs and give concrete examples of how to use the software supporting the users with up-to-date documentation and recipes 
 
Title kerchunk-builder 
Description A pipeline to convert NEtCDF file into Kerchunk data format. Kerchunk is a library that provides a unified way to represent a variety of chunked, compressed data formats (e.g. NetCDF/HDF5, GRIB2, TIFF, ), allowing efficient access to the data from traditional file systems or cloud object storage. Documentation available here: https://cedadev.github.io/kerchunk-builder/ 
Type Of Technology Software 
Year Produced 2024 
Impact The pipeline has been developed under the framework of EODH, and it has been used over a numerous of projects already (as such CCI) 
 
Description EO DataHub Project Outline for Industry Day, 13 March 2023 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Industry/Business
Results and Impact Presentation of the EO DataHub project to representatives from industry in order to engage them as suppliers for subsequent tenders for contracts to develop software and data services.
Year(s) Of Engagement Activity 2023
 
Description Presentation "EO DataHub Key Features" to the EO DataHub Users and Stakeholders Forum, 27 July 2023 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Industry/Business
Results and Impact A presentation on the EO DataHub project to a User and Stakeholder forum organised to disseminate the work of the project and engage with individuals and groups from the UK EO sector interested in being suppliers to its programme of work, or being users of the resulting software systems and data it provides.
Year(s) Of Engagement Activity 2023
 
Description Presentation on the EO DataHub project at the NCEO Conference 2023, 11-13 September, Leeds 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Other audiences
Results and Impact Present to the NCEO community the EO DataHub project describing its purpose and scope and how they can engage with its activities.
Year(s) Of Engagement Activity 2023
 
Description Presentation on the EO DataHub project to the Earth Observation Centre of Excellence group, 26 June 2023 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Presentation on the EO DataHub project to the EO Centre of Excellence a working group which brings together public sector users of Earth observation data. The purpose of the presentation was to raise awareness of the EO DataHub and the potential use and benefits to users of EO data in the public sector
Year(s) Of Engagement Activity 2023
 
Description Presentation on the EO DataHub project to the NCEO Staff Conference, London, 9 January 2024 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Other audiences
Results and Impact Provide an overview of the EO DataHub project to the NCEO community and explain to them how they can engage with this activity particular with respect to data outputs from the related EOCIS project.
Year(s) Of Engagement Activity 2024