System-wide Probabilistic Energy Forecasting

Lead Research Organisation: University of Strathclyde
Department Name: Electronic and Electrical Engineering

Abstract

The UK has binding targets to reduce carbon emission by 80% from 1990 levels by 2050. To achieve this, our energy systems are changing rapidly with a growing portion of electricity coming from renewable energy sources, and electrification of heating and transport. The result of this transition is an electricity system that is increasingly dependent on the weather: as well as having to manage variable amounts of power available from wind and solar resources, demand for electricity is becoming increasingly weather-dependent. Electricity network operators, generators and suppliers must rely on weather forecasts to plan their operations and ensure that supply meets demand, and they must do so in the knowledge that weather forecasts are imperfect, and therefore that future generation and demand uncertain.

This research will develop new energy forecasting methodologies to address the needs of the energy industry in this new paradigm. Energy forecasts are required for all weather-dependent elements of the electricity system, and their uncertainty must be quantified. Critically, there is a high degree of interdependence between uncertainty across the electricity system which must be captured to correctly characterise overall uncertainty. Furthermore, the precise nature of that interdependence will vary depending on specific weather conditions. The methodologies developed here will provide a framework for system-wide energy forecasting considering large-scale meteorological conditions, and provide decision-makers with valuable information about forecast uncertainty.

In addition, specific decision-support tools will be derived to condense voluminous and complex probabilistic forecast information into actionable analytical support. Tools to aid operational decision for power system operators, such as deciding how much back-up power to have available and how to manage constrains on the gird will be developed. Similarly, tools for generators and suppliers will be produced to enable more efficient participation in electricity markets. The overall objective of this work is to reduce the cost, and increase the reliability, of power systems with a high penetration of renewables.

Planned Impact

The growth of renewables in the UK and around the world is playing a major role in the global effort to mitigate the negative effects of climate change. As a result, power systems must adapt to function in a new paradigm where generation and demand are highly weather dependent, and where operators increasingly rely on weather forecasts, which are inherently imperfect. Equipping decision-makers with detailed forecast and uncertainty information, as this research will do, will enable more economic and reliable power system operation. As such, this work will have significant impact on the transition to low-carbon energy in the UK and globally.

Energy forecasts that quantify uncertainty and tools which convert that information in to actionable decision-support will contribute to continued integration of renewables while maintaining the high level of power system reliability and resilience expected of a developed economy. Furthermore, effective use of accurate uncertainty quantification will result in cost savings from reserve holding and electricity market operation which will translate to savings for energy consumers.

The project will have impact in the short-term resulting from advances in mathematical techniques for large-scale probabilistic forecasting that will have benefits across disciplines, such as econometrics and biostatistics. Over the longer term (3-4 years) the release of forecasting methodologies in an R package will enable researchers and industry to access novel forecasting tools in a timely way. Furthermore, training on use of the package will be provided for project partners and attendees at the final dissemination event. This will serve to up-skill researchers, enhance the effectiveness of users in the energy industry, and lead to new ideas for development and associated R&D funding. This research has already attracted significant interest from National Grid and ScottishPower who are keen to advance the use of probabilistic forecasting in their companies.

Numerous individuals will gain research and professional skills by being part of this research agenda. Both I and my collaborators will acquire skills in global engagement, communication and teamwork while expanding our research portfolios and skills. I will develop skills in meteorological and power system analysis while my collaborators will gain skills in statistical methods and energy domain knowledge. Research students will be up-skilled via interacting with this work through CDT mini-projects and PhDs, as will employees of industrial partners who will receive training in use of the methods produced by this work. This research lends itself to public engagement in understanding the necessity of science and engineering and their relevance to societal issues around energy and climate change. Through public UoS public engagement events and programmes run by the Glasgow Science Centre and Science Festival, which whom I have worked in the past, the importance of engineering research and STEM careers will be promoted.

Publications

10 25 50

Related Projects

Project Reference Relationship Related To Start End Award Value
EP/R023484/1 28/06/2018 31/07/2021 £312,934
EP/R023484/2 Transfer EP/R023484/1 01/08/2021 30/07/2022 £5,040
 
Description This work has developed new ways of forecasting Great Britain's electricity supply and demand in the days ahead. Such forecasts are essential to allow generators to schedule power production, and for National Grid to operate the transmission system, ensuring that supply and demand are in balance. In particular, this work has made it possible to accurately quantify the risk of very high or low demand, allowing the electricity market and National Grid to take appropriate action. Compared to current practice, these new methods provide confidence when forecast uncertainty is low, reducing the cost of operating the system, and identify times when risks may be high, ensuring appropriate actions can be taken to maintain reliable supply.
Exploitation Route The forecasting methods developed here are transferable to other situations where forecast uncertainty is important, particularly where low probability events and dependency between forecast variables is important.
Sectors Digital/Communication/Information Technologies (including Software),Energy

 
Description Analytical Middleware for Informed Distribution Networks (AMIDiNe)
Amount £703,091 (GBP)
Funding ID EP/S030131/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 04/2019 
End 05/2021
 
Description Network Innovation Allowance
Amount £400,000 (GBP)
Organisation National Grid UK 
Sector Private
Country United Kingdom
Start 11/2020 
End 07/2021
 
Description Supergen Energy Networks hub 2018
Amount £5,100,772 (GBP)
Funding ID EP/S00078X/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 09/2018 
End 09/2022
 
Title Data and Code for the EEM2020 Wind Power Forecasting Competition 
Description The competition data and code used by Team 12 to come second in the 2020 European Energy Market Conference Wind Power Forecasting Competition. The competition was organized and hosted by the EEM20 organizing team at KTH. The organizers thank Greenlytics for archiving the data for the competition. The weather data used for the competition has been acquired from met.no and the organizers acknowledge MET Norway for making it available. EEMwind2020_1.tar.gz is an R package containing the source code and data generated by Team 12. It may be installed from source using this file. EEM2020data.zip and EEM2020data2.zip contain the competition data. Weather forecasts are in netCDF format. Power and wind farm data are in csv files. EEM2020data should be placed in the same folder as the EEMwind2020 R package and the subfolders in *data2 should be moved to *data in order to run the scripts. 
Type Of Material Database/Collection of data 
Year Produced 2020 
Provided To Others? Yes  
URL https://pureportal.strath.ac.uk/en/datasets/02db89e9-64d1-4ded-9324-6d952ce35099
 
Title Supplementary material for: "Evaluation of Wind Power Forecasts - An up-to-date view" 
Description Code and data used to produce the examples in "Evaluation of Wind Power Forecasts - An up-to-date view" by Jakob W. Messner, Pierre Pinson, Jethro Browell, MathiasB Bjerregard, and Irene Schicker. A subset of data from GEFcom2014 is used (Task15_W_Zone1.csv), originally published here: Tao Hong, Pierre Pinson, Shu Fan, Hamidreza Zareipour, Alberto Troccoli, Rob J. Hyndman, "Probabilistic energy forecasting: Global Energy Forecasting Competition 2014 and beyond", International Journal of Forecasting,Volume 32, Issue 3,2016,Pages 896-913,ISSN 0169-2070,https://doi.org/10.1016/j.ijforecast.2016.02.001 This dataset is made available under the GNU General Public License v3.0 
Type Of Material Database/Collection of data 
Year Produced 2020 
Provided To Others? Yes  
URL https://pureportal.strath.ac.uk/en/datasets/3cf00c9b-e891-433e-afe3-732577aa74d2
 
Description Electricity Imbalance Price Forecasting 
Organisation Scottish Power Ltd
Country United Kingdom 
Sector Private 
PI Contribution I led the development of electricity imbalance price forecasting methodologies by my team, which included one full-time researcher working on this topic for 9 months. This work is a extension of the methods and findings of "System wide probabilistic energy forecasting", which focused on electricity supply and generation forecasting, to electricity market prices. An objective of SWPEF was to demonstrate novel methodologies in the context of electricity market participant, and this activity has been greatly enhanced by stronger collaboration with SSE and Scottish Power who have funded the extension in to price forecasting.
Collaborator Contribution SSE and Scottish Power provided expert insight and experience into the electricity market, including how to interpret complex market data, which was later leveraged in design of forecasting methods.
Impact We have delivered technical reports and a software tool implementing forecasting methods, which is presently being used in operation.
Start Year 2020
 
Description Electricity Imbalance Price Forecasting 
Organisation Scottish and Southern Energy (SSE)
Country United Kingdom 
Sector Private 
PI Contribution I led the development of electricity imbalance price forecasting methodologies by my team, which included one full-time researcher working on this topic for 9 months. This work is a extension of the methods and findings of "System wide probabilistic energy forecasting", which focused on electricity supply and generation forecasting, to electricity market prices. An objective of SWPEF was to demonstrate novel methodologies in the context of electricity market participant, and this activity has been greatly enhanced by stronger collaboration with SSE and Scottish Power who have funded the extension in to price forecasting.
Collaborator Contribution SSE and Scottish Power provided expert insight and experience into the electricity market, including how to interpret complex market data, which was later leveraged in design of forecasting methods.
Impact We have delivered technical reports and a software tool implementing forecasting methods, which is presently being used in operation.
Start Year 2020
 
Description Fassiolo 
Organisation University of Bristol
Country United Kingdom 
Sector Academic/University 
PI Contribution I have provided a vision for high-dimensional electricity (net) demand forecasting and applications of this capability. I have also provided a dataset to use in the development and testing of research ideas.
Collaborator Contribution Matteo Fassiolo has contributed expertise in computational statistics and ideas for sampling from multi-variate dependency models and modelling time-varying dependency.
Impact A joint paper has been submitted and we are developing a follow-on funding application. This is a multi-disciplinary collaboration between engineering (Browell) and statistics (Fassiolo).
Start Year 2019
 
Description National Grid ESO 
Organisation National Grid UK
Country United Kingdom 
Sector Private 
PI Contribution I have developed novel forecasting tools which provide more accurate and relevant information for power system operation than presently available. These include algorithms for wind power and electricity demand forecasting that are highly accurate and quantify remaining uncertainty. The latter part is important to allow decision makers to manage risk, for example holding reserve energy to compensate for forecast errors. My research has developed methods, demonstrated their value in case studies, and produced open source code to enable others to implement these methods.
Collaborator Contribution NGESO have provided data that is not publicly available as well as contextual knowledge and experience. They have also provided detailed use-cases based on current practice to to steer research direction and provide a basis for economic evaluation for forecast performance.
Impact One academic paper currently submitted, with accompanying data and computer code. Contributions to open source software package ProbCast (https://github.com/jbrowell/ProbCast/)
Start Year 2018
 
Description SSE Hydro Forecasting 
Organisation Scottish and Southern Energy (SSE)
Country United Kingdom 
Sector Private 
PI Contribution My team and I created a method for producing probabilistic forecasts, forecast which quantify the probability of future uncertain events, of the water resource in hydro power schemes. We developed statistical methods to convert state-of-the art numerical weather predictions into site-specific predictions up to six weeks ahead. This significantly increases forecast lead-time from current practice of only one week ahead. We estimated the value of this improved forecast information to be significant, increasing revenue by up to 5% for the same water resource through better management. This work is an extension to the original objectives of "System wide probabilistic energy forecasting", which focused on shorter lead-times; however, partnership this SSE identified extending forecast horizons for hydropower to be of particular interest, and funded a Research Associate to work with me to undertake additional work in this area.
Collaborator Contribution SSE provided operational data from hydro power schemes and their experience and operational practices. This data was used to develop and validate our forecasting methods, and as the basis for our economic evaluation of their benefit.
Impact We delivered three technical reports to SSE and have published one open-access journal article reporting our findings. This work is multi-disciplinary, bringing together meteorology and engineering.
Start Year 2019
 
Title ProbCast: an R package for probabilistic forecasting 
Description ProbCast is an R package to simply probabilistic forecasting. It provides a number of wrapper functions for established statistical learning techniques, and provides methods for forecast evaluation and visualisation. The ProbCast framework enables rapid development, testing and deployment of forecasting solutions. 
Type Of Technology Software 
Year Produced 2020 
Open Source License? Yes  
Impact ProbCast has stimulated engagement with other researchers and industry who would like to make use of it in their own research and operations. 
 
Description COMPASS CDT Hackathon 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Postgraduate students
Results and Impact r Jethro Browell, Research Fellow at the University of Strathclyde, and Dr Matteo Fasiolo, Lecturer at the University of Bristol, ran a regional electricity demand forecasting hackathon for students in the COMPASS Centre for Doctoral Training yesterday.
Visiting Research Fellow Dr Browell gave students an overview of how the Great Britain electricity transmission network has changed during the last decade, with particular focus on the consequences of the increased production from small-scale renewable sources, which appear as "negative demand".

Dr Fasiolo then introduced a dataset containing electricity demand and weather-related variables, such as wind speed and solar irradiation, from 14 regions covering the whole of Great Britain. He proposed an initial forecasting solution based on a simple Generalized Additive Model (GAM), which he used to forecast the demand in each region.

The hackathon started, with the "Jim" team being the first to propose an improved solution, based on a more sophisticated GAM model, which beat the initial GAM in terms of forecasting accuracy.

The "AGang" team then produced an even more sophisticated GAM, which took them to the top of the ranking. In the meantime, the "D&D" team was struggling to make their random forest work, and submitted a couple of poor forecasts. Toward the end of event, "AGang" produced a couple of improved GAM solutions, which further strengthened their lead.

While Dr Fasiolo and Dr Browell were wrapping up the event and preparing to award the winners, the "D&D" team caught everyone by surprise by submitting a forecast which beat all others by a margin, in terms of forecasting accuracy. Their random forest was far better than the GAMs at predicting demand in Scotland, where wind production is an important factor and the dynamics are quite different relative to the other regions.

Congratulations to the top three teams:

D&D: Doug Corbin and Dom Owens
AGang: Andrea Becsek, Alex Modell and Alessio Zakaria
Jim: Michael Whitehouse, Daniel Williams and Jake Spiteri
Winning team "D&D" said: "Given physical measurements, such as wind speeds and precipitation, as well as calendar data, we first performed a minor amount of feature engineering. Given the complex nature of the interactions between the variables, and large amount of data available, we opted to fit random forest models. These performed feature selection for us and provided some robustness from outlying observations.

"However, the models took a long time to fit. Despite parallelising the model fitting across the regions, we only just got our predictions in before the deadline. Thankfully, our model consistently outperformed the other approaches.

"Everyone taking part had a great time learning about the challenges of energy modelling, and we thrived under the pressure of friendly competition."

Dr Browell added: "Computational statistics and data science is driving innovation in the energy sector and the technologies they enable will play a huge role in the decarbonisation. I was pleased to be able to expose the COMPASS cohort to this application and hope that they will be inspired to apply their expertise to energy and climate problems in the future."
Year(s) Of Engagement Activity 2020
 
Description EEM 2020 Wind Power Forecasting Competition 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact I led a team including myself and three PhD students to second place in this international forecasting competition run by the European Electricity Markets Conference. We developed novel forecasting methods for the task and released these in open source code.
Year(s) Of Engagement Activity 2020
URL https://eem20.eu/forecasting-competition/
 
Description IEA Wind Tasks 36 and 51 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Active participation in an International Energy Agency Wind Task "Wind Power Forecasting" which aims to coordinate international research, interface with the industry, including developing standards, best-practice and training material. Key outcomes include academic papers (http://www.ieawindforecasting.dk/publications) and a recommended practice (http://www.ieawindforecasting.dk/Publications/RecommendedPractice) document on forecast evaluation, plus multiple presentations at international conferences.

I hosted a Workshop run in association with this IEA Task with 50 attendees, including representatives from UK and European utilities and research institutions) in January 2020 at the University of Strathclyde.
Year(s) Of Engagement Activity 2018,2019,2020,2021,2022
URL http://www.ieawindforecasting.dk
 
Description Lunchtime seminar for National Grid ESO via WebEx (during COVID-19 lockdown) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Industry/Business
Results and Impact Lunchtime talk/webinar given for National Grid ESO by Jethro Browell on Probabilistic Forecasting casting this in the context of their current challenges as the UK Energy System Operator. The event was attended by National Grid staff.
Year(s) Of Engagement Activity 2020
URL https://pureportal.strath.ac.uk/en/activities/system-wide-probabilistic-energy-forecasting
 
Description Member of IEEE Working Group on Energy Forecasting and Analytics 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Coordination of energy forecasting activity within the IEEE Power and Energy Society, including conference sessions, journal special issues and forecasting/data science competitions which are open to the public. In 2021 we are running a post-lockdown electricity load forecasting competition with over 150 participants.

IEEE Working Group on Energy Forecasting and Analytics
Scope:
-Energy forecasting:
-Forecasting objectives: load, renewable energy, price; individual consumer load, demand response, EV charging load, net load, wind power ramp; power, gas, heat, and cooling demands; reserve capacity, risk, network congestion;
-Forecasting algorithms: traditional regression, advanced machine learning, deep learning, transfer learning, ensemble learning, robust forecasting;
-Forecasting outputs: point forecasting, probabilistic forecasting, hierarchical forecasting, cost-oriented forecasting;
-Forecasting evaluation: Alternative loss functions for different forecasting objectives and different applications.
-Energy Analytics:
-Data preprocessing: outlier detection, data cleansing, feature selection, data compression;
-Behavior modeling: load profiling, energy theft detection, renewable energy spatiotemporal correlation analysis, pattern recognition, sensitivity analysis, load or renewable energy simulation;
-Applications: demand response implementation, data-driven pricing, bidding, and trading, topology identification, outage and risk management, privacy concerns.
Year(s) Of Engagement Activity 2018,2019,2020,2021
URL https://site.ieee.org/pes-psope/subcommittees/power-system-economics-subcommittee/working-group-on-e...
 
Description Next Generation Challenges in Energy-Climate Modelling 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact This workshop, which I co-organised, aimed to bridge the gab between energy modelling and weather & climate science. The workshop engaged over 70 participants in interactive discussions, resulting in a publication of findings and proposals for greater interaction between these disciplines.
Year(s) Of Engagement Activity 2020
URL https://research.reading.ac.uk/met-energy/next-generation-challenges-workshop/
 
Description UK Chapter of International Institute of Forecasters 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact This organisation runs the Quarterly Forecasting Forum, a quarterly workshop for forecasting practitioners, researchers and students to exchange knowledge. From 2019-2021 a have been secretary of the UK Chapter of the IIF supporting the organisation and running of events, and growing it's membership.
Year(s) Of Engagement Activity 2018,2019,2020
URL https://www.linkedin.com/groups/8839904/