Uncertainty analysis of hierarchical energy systems models: Models versus real energy systems
Lead Research Organisation:
Durham University
Department Name: Engineering and Computing Sciences
Abstract
Mathematical models are used widely in the planning and operation of energy systems, and in the development of public energy policy. The aim is to understand the impact of new policies, technologies and market operations. This is particularly significant at present due to the need to decarbonise energy systems over the coming decades, which is driving change in energy supply at a very rapid pace. Specific recent uses of large scale modelling studies in formal government policy Impact Assessments are the 2012 Energy Bill (which among other models uses economic modelling to project future generation investment under different policy options) and the 4th Carbon Budget (which provides the legal limit on GB carbon emissions from 2023 to 2027, and uses the UK MARKAL model for projecting evolution of the energy system under a given background scenario).
This project will study the relationship between mathematical and computer models of energy systems, and the real systems that they attempt to describe, with the purpose of enabling better model-based decisions in industry and government. Until such a relationship has been established between the model and the physical system that the model purports to represent, it is impossible to draw fully robust conclusions based on the model. This model/reality relationship will necessarily be probabilistic, expressing the degree to which uncertainty about the world can be resolved by careful use of the model. We will show, by a careful choice of exemplars, the ways in which such probabilistic relationships may be constructed for a wide range of energy systems models.
When we have linked the model and the physical system, the model can be embedded in a meaningful decision support system for choosing sensible future actions. This involves honest and careful assessment of all of the uncertainties involved in the planning process and consequent forecasting of the uncertainties associated with each possible planning choice. Using the selected exemplars, we will show how to replace current energy planning scenarios with a scrupulous uncertainty based guide to the consequences of current and future actions.
While much of this project involves specific exemplars, our intentions are general, namely to derive methodology which is widely applicable across the whole field of energy systems modelling and planning and therefore to show the potential for transformative analysis across the whole field. For this reason, our exemplars have been chosen to reflect general features common to a wide variety of energy models, which generally comprise systems of interconnected models which are each complex in their own right (e.g. a market model linking to an assessment of the engineering consequences of investment decisions, or a model describing interacting transmission and distribution networks). General methodology for the analysis of computer models will be tailored to the requirements of energy systems analysis. Further, some aspects, and in particular the development of effective uncertainty analysis for linked computer models, will have impact across modelling applications in many other fields.
Exemplars used will include studying the interaction between investment in generating capacity and the risks of supply shortages, the participation of resources embedded within distribution networks in the national energy market, and embedding a model of a particular sector of the energy economy within a model which projects the evolution of the whole energy economy. We will work on these exemplars in discussion with industrial collaborators, in order to identify how our methods must be designed and communicated in order for them to lead to eventual field application.
This project will study the relationship between mathematical and computer models of energy systems, and the real systems that they attempt to describe, with the purpose of enabling better model-based decisions in industry and government. Until such a relationship has been established between the model and the physical system that the model purports to represent, it is impossible to draw fully robust conclusions based on the model. This model/reality relationship will necessarily be probabilistic, expressing the degree to which uncertainty about the world can be resolved by careful use of the model. We will show, by a careful choice of exemplars, the ways in which such probabilistic relationships may be constructed for a wide range of energy systems models.
When we have linked the model and the physical system, the model can be embedded in a meaningful decision support system for choosing sensible future actions. This involves honest and careful assessment of all of the uncertainties involved in the planning process and consequent forecasting of the uncertainties associated with each possible planning choice. Using the selected exemplars, we will show how to replace current energy planning scenarios with a scrupulous uncertainty based guide to the consequences of current and future actions.
While much of this project involves specific exemplars, our intentions are general, namely to derive methodology which is widely applicable across the whole field of energy systems modelling and planning and therefore to show the potential for transformative analysis across the whole field. For this reason, our exemplars have been chosen to reflect general features common to a wide variety of energy models, which generally comprise systems of interconnected models which are each complex in their own right (e.g. a market model linking to an assessment of the engineering consequences of investment decisions, or a model describing interacting transmission and distribution networks). General methodology for the analysis of computer models will be tailored to the requirements of energy systems analysis. Further, some aspects, and in particular the development of effective uncertainty analysis for linked computer models, will have impact across modelling applications in many other fields.
Exemplars used will include studying the interaction between investment in generating capacity and the risks of supply shortages, the participation of resources embedded within distribution networks in the national energy market, and embedding a model of a particular sector of the energy economy within a model which projects the evolution of the whole energy economy. We will work on these exemplars in discussion with industrial collaborators, in order to identify how our methods must be designed and communicated in order for them to lead to eventual field application.
Planned Impact
The proposed research will benefit:
The government and electricity regulator. Improved systematic methods for analysing uncertainty in complex computer models will enable better public policy decisions to be made based on these models, helping deliver the required carbon reductions with improved cost and security of supply to customers.
The energy supply industry. The methods developed will enable better modelling-based decision making directly on the part of the industry; the Pathways to Impact document lists the various ways in which we will enable this transition of knowledge to the industry.
Energy customers. Ultimately the aim of enabling better model-based decision making by government and the energy industry is to enable the goals of decarbonisation and security of supply to be met at lower cost.
Users of complex computer models across the public, private and third sectors beyond the specific field of energy systems. The innovations in uncertainty analysis made in this project will find application wherever complex structures of linked models are used to support decisions. The methods developed bring potential benefits to any such organisation, resulting in better decision making and ultimately leading to more efficient and effective solutions for their citizens and customers.
The project will also have specific impact within the major industrial projects with which we will collaborate directly on modelling exemplars, notably the statutory Capacity Assessment project (feeding in to more robust design and operation of capacity mechanisms) and National Grid's work on transmission planning.
The government and electricity regulator. Improved systematic methods for analysing uncertainty in complex computer models will enable better public policy decisions to be made based on these models, helping deliver the required carbon reductions with improved cost and security of supply to customers.
The energy supply industry. The methods developed will enable better modelling-based decision making directly on the part of the industry; the Pathways to Impact document lists the various ways in which we will enable this transition of knowledge to the industry.
Energy customers. Ultimately the aim of enabling better model-based decision making by government and the energy industry is to enable the goals of decarbonisation and security of supply to be met at lower cost.
Users of complex computer models across the public, private and third sectors beyond the specific field of energy systems. The innovations in uncertainty analysis made in this project will find application wherever complex structures of linked models are used to support decisions. The methods developed bring potential benefits to any such organisation, resulting in better decision making and ultimately leading to more efficient and effective solutions for their citizens and customers.
The project will also have specific impact within the major industrial projects with which we will collaborate directly on modelling exemplars, notably the statutory Capacity Assessment project (feeding in to more robust design and operation of capacity mechanisms) and National Grid's work on transmission planning.
Organisations
- Durham University (Lead Research Organisation)
- Science and Technologies Facilities Council (STFC) (Collaboration)
- University College Dublin (Collaboration)
- Department for Business, Energy & Industrial Strategy (Collaboration)
- European Energy Research Alliance (EERA) (Collaboration)
- Department of Energy and Climate Change (Collaboration)
- National Grid UK (Collaboration)
Publications
Bialek J
(2016)
Benchmarking and Validation of Cascading Failure Analysis Tools
in IEEE Transactions on Power Systems
Cuffe P
(2018)
Data Visualization: The Signal and the Noise
in IEEE Potentials
Du H
(2021)
Beyond Strictly Proper Scoring Rules: The Importance of Being Local
in Weather and Forecasting
Smith L
(2020)
Designing Multimodel Applications with Surrogate Forecast Systems
in Monthly Weather Review
Wilson A
(2022)
Varying Coefficient Models and Design Choice for Bayes Linear Emulation of Complex Computer Models with Limited Model Evaluations
in SIAM/ASA Journal on Uncertainty Quantification
Wilson A
(2018)
Quantifying uncertainty in wholesale electricity price projections using Bayesian emulation of a generation investment model
in Sustainable Energy, Grids and Networks
Xu M
(2016)
Calibration and sensitivity analysis of long-term generation investment models using Bayesian emulation
in Sustainable Energy, Grids and Networks
Description | We believe this project is the only activity in the world using Bayesian model emulation withing uncertainty quantification in complex energy systems models. In general, investigating uncertainty in the relationship between outputs of a computer model and the real world system it purports to represent requires runnig the model multiple times - however if the model takes a long time to run then only a limited number of runs might be possible. This project has developed new methodology for making maximum use of a small number of runs - this was motivated by the fact that many energy models have very large numbers of inputs and outputs, and the number of runs ideally required increases with the numbers of inputs and outputs of the model. In the latter part of the project, work has been taken forward through a jointly funded researcher with the UK Centre for Energy Systems Integration. Applications have included building energy modelling and renewable energy system planning - in these topics, papers are either in preparation or review. This work has also led to relevant wider methodology developments, including in scoring rules for probabilistic forecasts. We have, more generally, demonstrated this class of methods for statistical uncertainty quantification on energy systems models for the first time - this has attracted the interest of many researchers and practitioners during talks and conferences. Future projects are discussed below. |
Exploitation Route | The first step on this is most likely to be adoption by academic groups with a more applied focus - while we are working with industry and government, and there may be opportunities for consultancy or impact projects on specific questions, the methodology developed might not make that direct leap from methodological academia to industry practice. Following (in autumn 2016) RA Amy Wilson spending 6 weeks on secondment to the government working on implementation of uncertainty quantification, we now have an Alan Turing Institute project on translating uncertainty government methods to wide practice, which will hopefully significantly accelerate non-academic impact. In addition, the Centre for Digital Built Britain has taken up thinking arising from this project in its research agenda, following a scoping network which was led by Chris Dent, and the two Investigators have co-written a further related white paper to be published by the Alan Turing Institute. |
Sectors | Energy Environment Government Democracy and Justice Transport |
Description | Ideas arising from this project have generated considerable interest in the government analyst community, including discussion with cross-government units who have oversight of modelling practice or who provide recommendations on development of practice. Specific applications of the relevant thinking are starting to develop, including decision analytic work linked to the electricity capacity market, a consulting report to Ofgem on decision analysis, and work with National Audit Office and local authories also connected to the Centre for Energy Systems Integration. |
Sector | Energy,Government, Democracy and Justice |
Impact Types | Policy & public services |
Description | IEEE |
Geographic Reach | Multiple continents/international |
Policy Influence Type | Influenced training of practitioners or researchers |
Impact | Chris Dent has the following roles within IEEE Power and Energy Society: - Vice Chair, Reliability, Risk and Probability Applications Subcommittee - Chair, LOLEWG (which shares experience between practical adequacy studies) - Chair, Task Force on Capacity Value of Solar Power - Chair, Working Group on Review of IEEE Standard 859 (transmission reliability data collection terminology) These contribute to development of industrial and academic knowledge and practice worldwide in the field of power system reliability analysis |
Description | Knowledge Transfer Secondment to BEIS |
Geographic Reach | National |
Policy Influence Type | Influenced training of practitioners or researchers |
Impact | Dr. Amy Wilson (postdoc) seconded to BEIS for 6 weeks to demonstrate prrof of concept implementation on uncertainty quantifiction methods. Result so far is improved knowledge and skills of BEIS analysts. |
Description | Submission to House of Lords Enquiry by Wilson, Goldstein and Dent |
Geographic Reach | National |
Policy Influence Type | Contribution to a national consultation/review |
URL | http://www.parliament.uk/resilience-of-electricity-infrastructure |
Description | Alan Turing Institute Partnership Project Scheme |
Amount | £403,000 (GBP) |
Organisation | Alan Turing Institute |
Sector | Academic/University |
Country | United Kingdom |
Start | 09/2018 |
End | 10/2018 |
Description | Alan Turing Institute Strategic Priorities Fund |
Amount | £64,000 (GBP) |
Organisation | Alan Turing Institute |
Sector | Academic/University |
Country | United Kingdom |
Start | 11/2018 |
End | 03/2019 |
Description | Centre for Digital Built Britain Research Network |
Amount | £46,990 (GBP) |
Organisation | Digital Built Britain |
Sector | Private |
Country | United Kingdom |
Start | 06/2018 |
End | 12/2018 |
Description | Durham University EPSRC IAA account |
Amount | £7,610 (GBP) |
Organisation | Durham University |
Sector | Academic/University |
Country | United Kingdom |
Start | 07/2016 |
End | 11/2016 |
Description | Linking Whole Energy System Models to the Real World |
Amount | £250,000 (GBP) |
Organisation | University of Edinburgh |
Sector | Academic/University |
Country | United Kingdom |
Start | 05/2020 |
End | 06/2023 |
Description | Local Heat and Energy Efficiency Strategies |
Amount | £30,000 (GBP) |
Organisation | Government of Scotland |
Sector | Public |
Country | United Kingdom |
Start | 09/2021 |
End | 03/2022 |
Description | National Digital Twin Programme CLimate Resilience Demonstrator |
Amount | £1,000,000 (GBP) |
Organisation | Department for Business, Energy & Industrial Strategy |
Sector | Public |
Country | United Kingdom |
Start | 03/2021 |
End | 03/2022 |
Description | DECC/Grid |
Organisation | Department of Energy and Climate Change |
Country | United Kingdom |
Sector | Public |
PI Contribution | Methodologies for statistical uncertainty quantification |
Collaborator Contribution | Provided the ddm model as an exemplar |
Impact | Contributed to numerous presentations and research discussions. In kind contribution is nominal - supply of a model which is used in real public policy decision making has a value which is almost unquantifiable. |
Start Year | 2015 |
Description | DECC/Grid |
Organisation | National Grid UK |
Country | United Kingdom |
Sector | Private |
PI Contribution | Methodologies for statistical uncertainty quantification |
Collaborator Contribution | Provided the ddm model as an exemplar |
Impact | Contributed to numerous presentations and research discussions. In kind contribution is nominal - supply of a model which is used in real public policy decision making has a value which is almost unquantifiable. |
Start Year | 2015 |
Description | ESIPP |
Organisation | University College Dublin |
Country | Ireland |
Sector | Academic/University |
PI Contribution | Provided letter of support. Providing ongoing input on statistical uncertainty quantification. |
Collaborator Contribution | Energy system model exemplars. Meteorological data. |
Impact | Multidisciplinary - engineering, maths, stats, meteorology |
Start Year | 2014 |
Description | Knowledge Transfer Secondment to BEIS |
Organisation | Department for Business, Energy & Industrial Strategy |
Country | United Kingdom |
Sector | Public |
PI Contribution | Knowledge transfer secondment of postdoc research Amy Wilson to BEIS, to implement proof of concept implemetation of methodology developed in academic work, and to scope future opportunities for collaboration. |
Collaborator Contribution | £750 cash for travel to meetings. Provision of runs from one of their computer models to use as exemplar in academic research. In-kind support in terms of time during the secondment. |
Impact | Presentations at various events, including tutorial day on energy systems planning under uncertainty. |
Start Year | 2015 |
Description | iiESI |
Organisation | European Energy Research Alliance (EERA) |
Country | Belgium |
Sector | Academic/University |
PI Contribution | Input to scoping documents of field of energy systems integration - EERA Joint Programme scheme of work, and International Institute for Energy Systems Integration green paper and other reports |
Collaborator Contribution | Overall coordination of activity in ESI, further intellectual input to reports, event organisation |
Impact | All disciplines relating to energy systems are involved. EERA JP in ESI scheme of work |
Start Year | 2014 |
Description | iiESI |
Organisation | Science and Technologies Facilities Council (STFC) |
Department | ISIS Neutron and Muon Source |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | Input to scoping documents of field of energy systems integration - EERA Joint Programme scheme of work, and International Institute for Energy Systems Integration green paper and other reports |
Collaborator Contribution | Overall coordination of activity in ESI, further intellectual input to reports, event organisation |
Impact | All disciplines relating to energy systems are involved. EERA JP in ESI scheme of work |
Start Year | 2014 |
Description | Amy Wilson presentation at ISBA |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Presentation at International Society for Bayesian Analysis on statistical model emulation with limited model runs |
Year(s) Of Engagement Activity | 2018 |
Description | Amy Wilson seminars to government |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Policymakers/politicians |
Results and Impact | Seminars to BEIS OR and energy groups |
Year(s) Of Engagement Activity | 2016 |
Description | BEIS Digital Twin steering group |
Form Of Engagement Activity | A formal working group, expert panel or dialogue |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Policymakers/politicians |
Results and Impact | Member of advisory group for a Business, Energy and Industrial Strategy-funded pilot project at Energy Systems Catapult |
Year(s) Of Engagement Activity | 2021,2022 |
Description | Blog article on energy modelling |
Form Of Engagement Activity | Engagement focused website, blog or social media channel |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Policymakers/politicians |
Results and Impact | Blog article on good modelling practice, as part of a University of Edinburgh series relating to the Scottish Government's energy strategy |
Year(s) Of Engagement Activity | 2017 |
URL | http://blogs.sps.ed.ac.uk/global-environment-society-academy/2017/05/22/energy-systems-modelling-mod... |
Description | Chris Dent and Amy Wilson presentations on capacity markets and uncertainty quantification at the Isaac Newton Institute Mathematics of Energy Systems programme |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Industry/Business |
Results and Impact | Presentations to mixed audience of researchers, industry and policy - part of origin of subsequent consultancy work on decision support for energy network planning. |
Year(s) Of Engagement Activity | 2019 |
Description | Chris Dent tutorial talk at N American Electricity Reliability Corporation Probabilistic Analysis Forum |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Industry/Business |
Results and Impact | Invited talk at major N American workshop on resource adequacy assessment |
Year(s) Of Engagement Activity | 2019 |
Description | GO Science masterclass |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Policymakers/politicians |
Results and Impact | Invited "Masterclass" by Chris Dent on modelling in public policy to the Government Office of Science Data Science Stream |
Year(s) Of Engagement Activity | 2018 |
Description | ICMS Energy Systems Workshop |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Industry/Business |
Results and Impact | Scoping workshop on mathematical science research needs in management of energy systems. Linked to Isaac Newton Institute programme on Mathematics of Energy Systems in 2019. |
Year(s) Of Engagement Activity | 2018 |
URL | http://www.icms.org.uk/energynetworks.php |
Description | Industry workshop linked to Isaac Newton Institute proposal |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Industry/Business |
Results and Impact | Co-organised industry scoping session associated with a proposal for an Energy Systems programme at the Isaac Newton Institute for Mathematical Sciences. |
Year(s) Of Engagement Activity | Pre-2006,2017 |
URL | https://iaciac.github.io/lobanet/events/ |
Description | Invited presentations in 2020-21 |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Industry/Business |
Results and Impact | Invited presentations to - International Conference on Probabilistic Methods Applied to Power Systems - Energy Regulators Regional Association - Cambridge Society of Edinburgh - European Union Agency for the Cooperation of Energy Regulators |
Year(s) Of Engagement Activity | 2020 |
Description | Invited presentations to UK industrial mathematics community |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | Dr Chris Dent has given invited presentations at: - Turing Gateway to Mathematics event on "Maths and Public Policy - Cities & Infrastructure", see http://www.turing-gateway.cam.ac.uk/mpci_mar2015-programme - KTN Industrial Mathematics community event, to stimulate interest in links between the mathematical sciences community and energy systems applications - KTN "Mathematics in Energy Systems" workshop, see https://www.eventbrite.co.uk/e/energy-catalyst-brokerage-event-london-tickets-19192257559?aff=erelexporg This outreach activity drew together experience from a number of different EPSRC projects. |
Year(s) Of Engagement Activity | 2015,2016 |
Description | KTN Energy Study Group, Edinburgh, December 2018 |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Industry/Business |
Results and Impact | Brainstorming session between mathematical scientists and five industry/government challenges |
Year(s) Of Engagement Activity | 2009,2018 |
URL | http://www.icms.org.uk/KTN_Energy_SG.php |
Description | Schools outreach by Amy Wilson |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Schools |
Results and Impact | - Speaker at UK mathematics trust maths circle event, May 2015, Hutcheson's school Glasgow. - International Women in Engineering day, 23/6/2016 - went to Whitworth Park school in Durham to talk to students about being a female in STEM. |
Year(s) Of Engagement Activity | 2015,2016 |
Description | Seminar and research discussions at NREL |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Presentation and research discussions at National Renewable Energy Laboratory in Denver, CO, developing common interests in resource adequacy assessment and energy system modelling. Particular topics under energy system modelling are the incorporation of operational constraints into planning models, and uncertainty quantification in plannign models, linking to the Future Conventional Power project. |
Year(s) Of Engagement Activity | 2019 |
Description | Seminar at Ofgem on capacity markets and decision support, by Chris Dent, Amy Wilson and Stan Zachary |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Policymakers/politicians |
Results and Impact | Seminar at energy regulator. Part of series of events leading to consultancy on decision support for network planning. |
Year(s) Of Engagement Activity | 2019 |
Description | Tutorial at MNIT Jaipur and seminar at IIT Delhi |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Postgraduate students |
Results and Impact | Gave tutorial on reliability analysis the day before HEAPD project workshop at MNIT Jaipur, and seminar on same subject at IIT Delhi the day after the HEAPD workshop. |
Year(s) Of Engagement Activity | 2016 |
Description | Tutorial day on Energy System Planning Under Uncertainty |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Other audiences |
Results and Impact | Tutorial day for academics and practitioners. Around 40 delegates including speakers. Aim was to translate methods from mathematical science to application disciplines. Slides and videos of talks available on website, see URL. Sponsored by Hubnet and CESI. Other grants listed indicate source of methodology presented in one or more talks, or payment of travel for a speaker. |
Year(s) Of Engagement Activity | 2016 |
URL | http://icms.org.uk/workshops/energytutorialday |
Description | Visit to industry colleagues in California to broaden impact links |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Industry/Business |
Results and Impact | Visit to colleagues in California, including CA Independent System Operator, Public Utilities Commission, Stanford Research International, Electric Power Research Institute. Aim is to broaden impact of EPSRC work on uncertainty quantification in complex computer models, including discussion of future collaboration and industry projects. |
Year(s) Of Engagement Activity | 2017 |
Description | Workshop on "Modelling in public policy" |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Policymakers/politicians |
Results and Impact | Scoping workshop on research needs in modelling to support public policy, report at http://hubnet.org.uk/filebyid/1130/Modelling_in_Public_Policy.pdf |
Year(s) Of Engagement Activity | 2017 |