System-wide Probabilistic Energy Forecasting
Lead Research Organisation:
University of Strathclyde
Department Name: Electronic and Electrical Engineering
Abstract
The UK has binding targets to reduce carbon emission by 80% from 1990 levels by 2050. To achieve this, our energy systems are changing rapidly with a growing portion of electricity coming from renewable energy sources, and electrification of heating and transport. The result of this transition is an electricity system that is increasingly dependent on the weather: as well as having to manage variable amounts of power available from wind and solar resources, demand for electricity is becoming increasingly weather-dependent. Electricity network operators, generators and suppliers must rely on weather forecasts to plan their operations and ensure that supply meets demand, and they must do so in the knowledge that weather forecasts are imperfect, and therefore that future generation and demand uncertain.
This research will develop new energy forecasting methodologies to address the needs of the energy industry in this new paradigm. Energy forecasts are required for all weather-dependent elements of the electricity system, and their uncertainty must be quantified. Critically, there is a high degree of interdependence between uncertainty across the electricity system which must be captured to correctly characterise overall uncertainty. Furthermore, the precise nature of that interdependence will vary depending on specific weather conditions. The methodologies developed here will provide a framework for system-wide energy forecasting considering large-scale meteorological conditions, and provide decision-makers with valuable information about forecast uncertainty.
In addition, specific decision-support tools will be derived to condense voluminous and complex probabilistic forecast information into actionable analytical support. Tools to aid operational decision for power system operators, such as deciding how much back-up power to have available and how to manage constrains on the gird will be developed. Similarly, tools for generators and suppliers will be produced to enable more efficient participation in electricity markets. The overall objective of this work is to reduce the cost, and increase the reliability, of power systems with a high penetration of renewables.
This research will develop new energy forecasting methodologies to address the needs of the energy industry in this new paradigm. Energy forecasts are required for all weather-dependent elements of the electricity system, and their uncertainty must be quantified. Critically, there is a high degree of interdependence between uncertainty across the electricity system which must be captured to correctly characterise overall uncertainty. Furthermore, the precise nature of that interdependence will vary depending on specific weather conditions. The methodologies developed here will provide a framework for system-wide energy forecasting considering large-scale meteorological conditions, and provide decision-makers with valuable information about forecast uncertainty.
In addition, specific decision-support tools will be derived to condense voluminous and complex probabilistic forecast information into actionable analytical support. Tools to aid operational decision for power system operators, such as deciding how much back-up power to have available and how to manage constrains on the gird will be developed. Similarly, tools for generators and suppliers will be produced to enable more efficient participation in electricity markets. The overall objective of this work is to reduce the cost, and increase the reliability, of power systems with a high penetration of renewables.
Planned Impact
The growth of renewables in the UK and around the world is playing a major role in the global effort to mitigate the negative effects of climate change. As a result, power systems must adapt to function in a new paradigm where generation and demand are highly weather dependent, and where operators increasingly rely on weather forecasts, which are inherently imperfect. Equipping decision-makers with detailed forecast and uncertainty information, as this research will do, will enable more economic and reliable power system operation. As such, this work will have significant impact on the transition to low-carbon energy in the UK and globally.
Energy forecasts that quantify uncertainty and tools which convert that information in to actionable decision-support will contribute to continued integration of renewables while maintaining the high level of power system reliability and resilience expected of a developed economy. Furthermore, effective use of accurate uncertainty quantification will result in cost savings from reserve holding and electricity market operation which will translate to savings for energy consumers.
The project will have impact in the short-term resulting from advances in mathematical techniques for large-scale probabilistic forecasting that will have benefits across disciplines, such as econometrics and biostatistics. Over the longer term (3-4 years) the release of forecasting methodologies in an R package will enable researchers and industry to access novel forecasting tools in a timely way. Furthermore, training on use of the package will be provided for project partners and attendees at the final dissemination event. This will serve to up-skill researchers, enhance the effectiveness of users in the energy industry, and lead to new ideas for development and associated R&D funding. This research has already attracted significant interest from National Grid and ScottishPower who are keen to advance the use of probabilistic forecasting in their companies.
Numerous individuals will gain research and professional skills by being part of this research agenda. Both I and my collaborators will acquire skills in global engagement, communication and teamwork while expanding our research portfolios and skills. I will develop skills in meteorological and power system analysis while my collaborators will gain skills in statistical methods and energy domain knowledge. Research students will be up-skilled via interacting with this work through CDT mini-projects and PhDs, as will employees of industrial partners who will receive training in use of the methods produced by this work. This research lends itself to public engagement in understanding the necessity of science and engineering and their relevance to societal issues around energy and climate change. Through public UoS public engagement events and programmes run by the Glasgow Science Centre and Science Festival, which whom I have worked in the past, the importance of engineering research and STEM careers will be promoted.
Energy forecasts that quantify uncertainty and tools which convert that information in to actionable decision-support will contribute to continued integration of renewables while maintaining the high level of power system reliability and resilience expected of a developed economy. Furthermore, effective use of accurate uncertainty quantification will result in cost savings from reserve holding and electricity market operation which will translate to savings for energy consumers.
The project will have impact in the short-term resulting from advances in mathematical techniques for large-scale probabilistic forecasting that will have benefits across disciplines, such as econometrics and biostatistics. Over the longer term (3-4 years) the release of forecasting methodologies in an R package will enable researchers and industry to access novel forecasting tools in a timely way. Furthermore, training on use of the package will be provided for project partners and attendees at the final dissemination event. This will serve to up-skill researchers, enhance the effectiveness of users in the energy industry, and lead to new ideas for development and associated R&D funding. This research has already attracted significant interest from National Grid and ScottishPower who are keen to advance the use of probabilistic forecasting in their companies.
Numerous individuals will gain research and professional skills by being part of this research agenda. Both I and my collaborators will acquire skills in global engagement, communication and teamwork while expanding our research portfolios and skills. I will develop skills in meteorological and power system analysis while my collaborators will gain skills in statistical methods and energy domain knowledge. Research students will be up-skilled via interacting with this work through CDT mini-projects and PhDs, as will employees of industrial partners who will receive training in use of the methods produced by this work. This research lends itself to public engagement in understanding the necessity of science and engineering and their relevance to societal issues around energy and climate change. Through public UoS public engagement events and programmes run by the Glasgow Science Centre and Science Festival, which whom I have worked in the past, the importance of engineering research and STEM careers will be promoted.
Organisations
- University of Strathclyde, United Kingdom (Lead Research Organisation)
- University of Bristol, United Kingdom (Collaboration)
- Scottish Power Ltd (Collaboration)
- Scottish and Southern Energy (SSE) (Collaboration)
- National Grid UK (Collaboration)
- Scottish and Southern Energy SSE plc, United Kingdom (Project Partner)
- Scottish Power Renewables Ltd, United Kingdom (Project Partner)
- National Grid PLC, United Kingdom (Project Partner)
- University of Glasgow, United Kingdom (Fellow)
Publications

Bloomfield H
(2021)
The Importance of Weather and Climate to Energy Systems: A Workshop on Next Generation Challenges in Energy-Climate Modeling
in Bulletin of the American Meteorological Society

Browell J
(2021)
Probabilistic Forecasting of Regional Net-Load With Conditional Extremes and Gridded NWP
in IEEE Transactions on Smart Grid


Edmunds C
(2019)
On the participation of wind energy in response and reserve markets in Great Britain and Spain
in Renewable and Sustainable Energy Reviews

Farrokhabadi M
(2022)
Day-Ahead Electricity Demand Forecasting Competition: Post-COVID Paradigm
in IEEE Open Access Journal of Power and Energy

Graham R
(2022)
The application of sub-seasonal to seasonal (S2S) predictions for hydropower forecasting
in Meteorological Applications

Heylen E
(2021)
Probabilistic Day-ahead Inertia Forecasting
in IEEE Transactions on Power Systems

Messner J
(2020)
Evaluation of wind power forecasts-An up-to-date view
in Wind Energy

Sweeney C
(2019)
The future of forecasting for renewable energy
in WIREs Energy and Environment
Related Projects
Project Reference | Relationship | Related To | Start | End | Award Value |
---|---|---|---|---|---|
EP/R023484/1 | 28/06/2018 | 31/07/2021 | £312,934 | ||
EP/R023484/2 | Transfer | EP/R023484/1 | 01/08/2021 | 30/07/2022 | £5,040 |
Description | This work has developed new ways of forecasting Great Britain's electricity supply and demand in the days ahead. Such forecasts are essential to allow generators to schedule power production, and for National Grid to operate the transmission system, ensuring that supply and demand are in balance. In particular, this work has made it possible to accurately quantify the risk of very high or low demand, allowing the electricity market and National Grid to take appropriate action. Compared to current practice, these new methods provide confidence when forecast uncertainty is low, reducing the cost of operating the system, and identify times when risks may be high, ensuring appropriate actions can be taken to maintain reliable supply. |
Exploitation Route | The forecasting methods developed here are transferable to other situations where forecast uncertainty is important, particularly where low probability events and dependency between forecast variables is important. |
Sectors | Digital/Communication/Information Technologies (including Software),Energy |
Description | Analytical Middleware for Informed Distribution Networks (AMIDiNe) |
Amount | £703,091 (GBP) |
Funding ID | EP/S030131/1 |
Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
Sector | Public |
Country | United Kingdom |
Start | 04/2019 |
End | 05/2021 |
Description | Network Innovation Allowance |
Amount | £400,000 (GBP) |
Organisation | National Grid UK |
Sector | Private |
Country | United Kingdom |
Start | 11/2020 |
End | 07/2021 |
Description | Supergen Energy Networks hub 2018 |
Amount | £5,100,772 (GBP) |
Funding ID | EP/S00078X/1 |
Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
Sector | Public |
Country | United Kingdom |
Start | 09/2018 |
End | 09/2022 |
Title | Data and Code for the EEM2020 Wind Power Forecasting Competition |
Description | The competition data and code used by Team 12 to come second in the 2020 European Energy Market Conference Wind Power Forecasting Competition. The competition was organized and hosted by the EEM20 organizing team at KTH. The organizers thank Greenlytics for archiving the data for the competition. The weather data used for the competition has been acquired from met.no and the organizers acknowledge MET Norway for making it available. EEMwind2020_1.tar.gz is an R package containing the source code and data generated by Team 12. It may be installed from source using this file. EEM2020data.zip and EEM2020data2.zip contain the competition data. Weather forecasts are in netCDF format. Power and wind farm data are in csv files. EEM2020data should be placed in the same folder as the EEMwind2020 R package and the subfolders in *data2 should be moved to *data in order to run the scripts. |
Type Of Material | Database/Collection of data |
Year Produced | 2020 |
Provided To Others? | Yes |
URL | https://pureportal.strath.ac.uk/en/datasets/02db89e9-64d1-4ded-9324-6d952ce35099 |
Title | Supplementary material for: "Evaluation of Wind Power Forecasts - An up-to-date view" |
Description | Code and data used to produce the examples in "Evaluation of Wind Power Forecasts - An up-to-date view" by Jakob W. Messner, Pierre Pinson, Jethro Browell, MathiasB Bjerregard, and Irene Schicker. A subset of data from GEFcom2014 is used (Task15_W_Zone1.csv), originally published here: Tao Hong, Pierre Pinson, Shu Fan, Hamidreza Zareipour, Alberto Troccoli, Rob J. Hyndman, "Probabilistic energy forecasting: Global Energy Forecasting Competition 2014 and beyond", International Journal of Forecasting,Volume 32, Issue 3,2016,Pages 896-913,ISSN 0169-2070,https://doi.org/10.1016/j.ijforecast.2016.02.001 This dataset is made available under the GNU General Public License v3.0 |
Type Of Material | Database/Collection of data |
Year Produced | 2020 |
Provided To Others? | Yes |
URL | https://pureportal.strath.ac.uk/en/datasets/3cf00c9b-e891-433e-afe3-732577aa74d2 |
Description | Electricity Imbalance Price Forecasting |
Organisation | Scottish Power Ltd |
Country | United Kingdom |
Sector | Private |
PI Contribution | I led the development of electricity imbalance price forecasting methodologies by my team, which included one full-time researcher working on this topic for 9 months. This work is a extension of the methods and findings of "System wide probabilistic energy forecasting", which focused on electricity supply and generation forecasting, to electricity market prices. An objective of SWPEF was to demonstrate novel methodologies in the context of electricity market participant, and this activity has been greatly enhanced by stronger collaboration with SSE and Scottish Power who have funded the extension in to price forecasting. |
Collaborator Contribution | SSE and Scottish Power provided expert insight and experience into the electricity market, including how to interpret complex market data, which was later leveraged in design of forecasting methods. |
Impact | We have delivered technical reports and a software tool implementing forecasting methods, which is presently being used in operation. |
Start Year | 2020 |
Description | Electricity Imbalance Price Forecasting |
Organisation | Scottish and Southern Energy (SSE) |
Country | United Kingdom |
Sector | Private |
PI Contribution | I led the development of electricity imbalance price forecasting methodologies by my team, which included one full-time researcher working on this topic for 9 months. This work is a extension of the methods and findings of "System wide probabilistic energy forecasting", which focused on electricity supply and generation forecasting, to electricity market prices. An objective of SWPEF was to demonstrate novel methodologies in the context of electricity market participant, and this activity has been greatly enhanced by stronger collaboration with SSE and Scottish Power who have funded the extension in to price forecasting. |
Collaborator Contribution | SSE and Scottish Power provided expert insight and experience into the electricity market, including how to interpret complex market data, which was later leveraged in design of forecasting methods. |
Impact | We have delivered technical reports and a software tool implementing forecasting methods, which is presently being used in operation. |
Start Year | 2020 |
Description | Fassiolo |
Organisation | University of Bristol |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | I have provided a vision for high-dimensional electricity (net) demand forecasting and applications of this capability. I have also provided a dataset to use in the development and testing of research ideas. |
Collaborator Contribution | Matteo Fassiolo has contributed expertise in computational statistics and ideas for sampling from multi-variate dependency models and modelling time-varying dependency. |
Impact | A joint paper has been submitted and we are developing a follow-on funding application. This is a multi-disciplinary collaboration between engineering (Browell) and statistics (Fassiolo). |
Start Year | 2019 |
Description | National Grid ESO |
Organisation | National Grid UK |
Country | United Kingdom |
Sector | Private |
PI Contribution | I have developed novel forecasting tools which provide more accurate and relevant information for power system operation than presently available. These include algorithms for wind power and electricity demand forecasting that are highly accurate and quantify remaining uncertainty. The latter part is important to allow decision makers to manage risk, for example holding reserve energy to compensate for forecast errors. My research has developed methods, demonstrated their value in case studies, and produced open source code to enable others to implement these methods. |
Collaborator Contribution | NGESO have provided data that is not publicly available as well as contextual knowledge and experience. They have also provided detailed use-cases based on current practice to to steer research direction and provide a basis for economic evaluation for forecast performance. |
Impact | One academic paper currently submitted, with accompanying data and computer code. Contributions to open source software package ProbCast (https://github.com/jbrowell/ProbCast/) |
Start Year | 2018 |
Description | SSE Hydro Forecasting |
Organisation | Scottish and Southern Energy (SSE) |
Country | United Kingdom |
Sector | Private |
PI Contribution | My team and I created a method for producing probabilistic forecasts, forecast which quantify the probability of future uncertain events, of the water resource in hydro power schemes. We developed statistical methods to convert state-of-the art numerical weather predictions into site-specific predictions up to six weeks ahead. This significantly increases forecast lead-time from current practice of only one week ahead. We estimated the value of this improved forecast information to be significant, increasing revenue by up to 5% for the same water resource through better management. This work is an extension to the original objectives of "System wide probabilistic energy forecasting", which focused on shorter lead-times; however, partnership this SSE identified extending forecast horizons for hydropower to be of particular interest, and funded a Research Associate to work with me to undertake additional work in this area. |
Collaborator Contribution | SSE provided operational data from hydro power schemes and their experience and operational practices. This data was used to develop and validate our forecasting methods, and as the basis for our economic evaluation of their benefit. |
Impact | We delivered three technical reports to SSE and have published one open-access journal article reporting our findings. This work is multi-disciplinary, bringing together meteorology and engineering. |
Start Year | 2019 |
Title | ProbCast: an R package for probabilistic forecasting |
Description | ProbCast is an R package to simply probabilistic forecasting. It provides a number of wrapper functions for established statistical learning techniques, and provides methods for forecast evaluation and visualisation. The ProbCast framework enables rapid development, testing and deployment of forecasting solutions. |
Type Of Technology | Software |
Year Produced | 2020 |
Open Source License? | Yes |
Impact | ProbCast has stimulated engagement with other researchers and industry who would like to make use of it in their own research and operations. |
Description | COMPASS CDT Hackathon |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Postgraduate students |
Results and Impact | r Jethro Browell, Research Fellow at the University of Strathclyde, and Dr Matteo Fasiolo, Lecturer at the University of Bristol, ran a regional electricity demand forecasting hackathon for students in the COMPASS Centre for Doctoral Training yesterday. Visiting Research Fellow Dr Browell gave students an overview of how the Great Britain electricity transmission network has changed during the last decade, with particular focus on the consequences of the increased production from small-scale renewable sources, which appear as "negative demand". Dr Fasiolo then introduced a dataset containing electricity demand and weather-related variables, such as wind speed and solar irradiation, from 14 regions covering the whole of Great Britain. He proposed an initial forecasting solution based on a simple Generalized Additive Model (GAM), which he used to forecast the demand in each region. The hackathon started, with the "Jim" team being the first to propose an improved solution, based on a more sophisticated GAM model, which beat the initial GAM in terms of forecasting accuracy. The "AGang" team then produced an even more sophisticated GAM, which took them to the top of the ranking. In the meantime, the "D&D" team was struggling to make their random forest work, and submitted a couple of poor forecasts. Toward the end of event, "AGang" produced a couple of improved GAM solutions, which further strengthened their lead. While Dr Fasiolo and Dr Browell were wrapping up the event and preparing to award the winners, the "D&D" team caught everyone by surprise by submitting a forecast which beat all others by a margin, in terms of forecasting accuracy. Their random forest was far better than the GAMs at predicting demand in Scotland, where wind production is an important factor and the dynamics are quite different relative to the other regions. Congratulations to the top three teams: D&D: Doug Corbin and Dom Owens AGang: Andrea Becsek, Alex Modell and Alessio Zakaria Jim: Michael Whitehouse, Daniel Williams and Jake Spiteri Winning team "D&D" said: "Given physical measurements, such as wind speeds and precipitation, as well as calendar data, we first performed a minor amount of feature engineering. Given the complex nature of the interactions between the variables, and large amount of data available, we opted to fit random forest models. These performed feature selection for us and provided some robustness from outlying observations. "However, the models took a long time to fit. Despite parallelising the model fitting across the regions, we only just got our predictions in before the deadline. Thankfully, our model consistently outperformed the other approaches. "Everyone taking part had a great time learning about the challenges of energy modelling, and we thrived under the pressure of friendly competition." Dr Browell added: "Computational statistics and data science is driving innovation in the energy sector and the technologies they enable will play a huge role in the decarbonisation. I was pleased to be able to expose the COMPASS cohort to this application and hope that they will be inspired to apply their expertise to energy and climate problems in the future." |
Year(s) Of Engagement Activity | 2020 |
Description | EEM 2020 Wind Power Forecasting Competition |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | I led a team including myself and three PhD students to second place in this international forecasting competition run by the European Electricity Markets Conference. We developed novel forecasting methods for the task and released these in open source code. |
Year(s) Of Engagement Activity | 2020 |
URL | https://eem20.eu/forecasting-competition/ |
Description | IEA Wind Tasks 36 and 51 |
Form Of Engagement Activity | A formal working group, expert panel or dialogue |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Active participation in an International Energy Agency Wind Task "Wind Power Forecasting" which aims to coordinate international research, interface with the industry, including developing standards, best-practice and training material. Key outcomes include academic papers (http://www.ieawindforecasting.dk/publications) and a recommended practice (http://www.ieawindforecasting.dk/Publications/RecommendedPractice) document on forecast evaluation, plus multiple presentations at international conferences. I hosted a Workshop run in association with this IEA Task with 50 attendees, including representatives from UK and European utilities and research institutions) in January 2020 at the University of Strathclyde. |
Year(s) Of Engagement Activity | 2018,2019,2020,2021,2022 |
URL | http://www.ieawindforecasting.dk |
Description | Lunchtime seminar for National Grid ESO via WebEx (during COVID-19 lockdown) |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Industry/Business |
Results and Impact | Lunchtime talk/webinar given for National Grid ESO by Jethro Browell on Probabilistic Forecasting casting this in the context of their current challenges as the UK Energy System Operator. The event was attended by National Grid staff. |
Year(s) Of Engagement Activity | 2020 |
URL | https://pureportal.strath.ac.uk/en/activities/system-wide-probabilistic-energy-forecasting |
Description | Member of IEEE Working Group on Energy Forecasting and Analytics |
Form Of Engagement Activity | A formal working group, expert panel or dialogue |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Coordination of energy forecasting activity within the IEEE Power and Energy Society, including conference sessions, journal special issues and forecasting/data science competitions which are open to the public. In 2021 we are running a post-lockdown electricity load forecasting competition with over 150 participants. IEEE Working Group on Energy Forecasting and Analytics Scope: -Energy forecasting: -Forecasting objectives: load, renewable energy, price; individual consumer load, demand response, EV charging load, net load, wind power ramp; power, gas, heat, and cooling demands; reserve capacity, risk, network congestion; -Forecasting algorithms: traditional regression, advanced machine learning, deep learning, transfer learning, ensemble learning, robust forecasting; -Forecasting outputs: point forecasting, probabilistic forecasting, hierarchical forecasting, cost-oriented forecasting; -Forecasting evaluation: Alternative loss functions for different forecasting objectives and different applications. -Energy Analytics: -Data preprocessing: outlier detection, data cleansing, feature selection, data compression; -Behavior modeling: load profiling, energy theft detection, renewable energy spatiotemporal correlation analysis, pattern recognition, sensitivity analysis, load or renewable energy simulation; -Applications: demand response implementation, data-driven pricing, bidding, and trading, topology identification, outage and risk management, privacy concerns. |
Year(s) Of Engagement Activity | 2018,2019,2020,2021 |
URL | https://site.ieee.org/pes-psope/subcommittees/power-system-economics-subcommittee/working-group-on-e... |
Description | Next Generation Challenges in Energy-Climate Modelling |
Form Of Engagement Activity | A formal working group, expert panel or dialogue |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | This workshop, which I co-organised, aimed to bridge the gab between energy modelling and weather & climate science. The workshop engaged over 70 participants in interactive discussions, resulting in a publication of findings and proposals for greater interaction between these disciplines. |
Year(s) Of Engagement Activity | 2020 |
URL | https://research.reading.ac.uk/met-energy/next-generation-challenges-workshop/ |
Description | UK Chapter of International Institute of Forecasters |
Form Of Engagement Activity | A formal working group, expert panel or dialogue |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | This organisation runs the Quarterly Forecasting Forum, a quarterly workshop for forecasting practitioners, researchers and students to exchange knowledge. From 2019-2021 a have been secretary of the UK Chapter of the IIF supporting the organisation and running of events, and growing it's membership. |
Year(s) Of Engagement Activity | 2018,2019,2020 |
URL | https://www.linkedin.com/groups/8839904/ |