TECHNOLOGY MARKET ASSESSMENT AND FOLLOW ON FUND APPLICATION SUPPORT FOR THE SAFE SOFTWARE TOOL
Lead Research Organisation:
University of Bristol
Department Name: Civil Engineering
Abstract
Complex numerical models are core tools that are used in many industries. Their uses range from the design of complex mechanical structures such as windmills, over the analysis of windstorm risk across Europe for the insurance industry, to water management models used for developing sustainable use policies for the Thames region. Calibration and evaluation is difficult for these typically very complex and non-linear models, for which informal approaches such as trial-and-error, visual analysis or expert judgement quickly become unfeasible. More stringent quality criteria are increasingly demanded or even required for numerical models, which are used for design and policy development. Such analyses often also have to include uncertainty propagation through the model. Unless the uncertainty in the predictions produced by these models can be quantified and characterised they will not be suitable as a basis for decision-making. Specific tools are required to analyse, understand and convey these uncertainties, their sources and their effects on the model outputs. Such tools need to be easy to use and communicate to specialists and non-specialists alike.
Within the NERC-funded CREDIBLE Project, we have developed a software toolbox named SAFE (Sensitivity Analysis For Everybody). SAFE is a numerical tool to perform Global Sensitivity Analysis (GSA), a set of statistical techniques providing a quantitative and structured approach to the development and analysis of numerical models. It implements a comprehensive suite of GSA methods and visualisation tools, combined in an extensively tested and user-friendly Toolbox. Since the CREDIBLE Project was specifically aimed at developing generic tools across a wide range of natural hazard domains, the Toolbox is well suited for further development into a completely generic software tool. The commercial potential of such GSA software product would extend to virtually any industrial domain where numerical models are used, from manufacturing companies to environmental consultancy or insurance.
Although the GSA approaches implemented in SAFE are well established, validated and published, they are not readily accessible to a non-specialist audience and cannot quickly be implemented for new models. Existing software tools are either "bare" code, which requires significant programing expertise to implement, or very specialised tools that focus on a specific methodology and are not more widely applicable. The SAFE toolbox overcomes both these drawbacks, and we hope to develop it into a user-friendly, easily adaptable tool that can be used in a range of sectors.
In our Pathfinder project, we intend to (i) investigate the market sectors that will be able to benefit most from the toolbox and therefore could be the most attractive market sectors to target for early adoption; (ii) to gain input from end-users on the technical and support features needed for the tool so to inform the technical development of the commercial version of SAFE; and (iii) to identify the best route to market for the product.
Keywords: Numerical Modelling, Sensitivity Analysis, Uncertainty Analysis, Calibration, Validation, Evaluation
Stakeholders: Industry, Government, Policymakers, Academia
Within the NERC-funded CREDIBLE Project, we have developed a software toolbox named SAFE (Sensitivity Analysis For Everybody). SAFE is a numerical tool to perform Global Sensitivity Analysis (GSA), a set of statistical techniques providing a quantitative and structured approach to the development and analysis of numerical models. It implements a comprehensive suite of GSA methods and visualisation tools, combined in an extensively tested and user-friendly Toolbox. Since the CREDIBLE Project was specifically aimed at developing generic tools across a wide range of natural hazard domains, the Toolbox is well suited for further development into a completely generic software tool. The commercial potential of such GSA software product would extend to virtually any industrial domain where numerical models are used, from manufacturing companies to environmental consultancy or insurance.
Although the GSA approaches implemented in SAFE are well established, validated and published, they are not readily accessible to a non-specialist audience and cannot quickly be implemented for new models. Existing software tools are either "bare" code, which requires significant programing expertise to implement, or very specialised tools that focus on a specific methodology and are not more widely applicable. The SAFE toolbox overcomes both these drawbacks, and we hope to develop it into a user-friendly, easily adaptable tool that can be used in a range of sectors.
In our Pathfinder project, we intend to (i) investigate the market sectors that will be able to benefit most from the toolbox and therefore could be the most attractive market sectors to target for early adoption; (ii) to gain input from end-users on the technical and support features needed for the tool so to inform the technical development of the commercial version of SAFE; and (iii) to identify the best route to market for the product.
Keywords: Numerical Modelling, Sensitivity Analysis, Uncertainty Analysis, Calibration, Validation, Evaluation
Stakeholders: Industry, Government, Policymakers, Academia
Planned Impact
The Pathfinder project is intended to provide comprehensive insight into the market needs and requirements for GSA analysis and the tools that can support such work. We will use this insight to design a Follow on Fund (FoF) project to develop the SAFE toolbox so that it can deliver these needs. The FoF project is likely to involve improvements to the user interface; development of new functionalities to meet the specific requirements of commercial users; development of appropriate training and support; and the establishment of a commercialisation pathway including appropriate intellectual property protection to enable further investment.
The outcome of the FoF project would be a customer-focused toolkit, which is fit for purpose and meets the needs of a diverse range of users in different market sectors.
For example, just one of these sectors, environmental consultancy, is a significant global market which continues to grow, with an estimated value of $27.4bn in 2014, and is predicted to expand further over the next five years. The revenues from successful exploitation of the tool within one or more of these markets can be expected to provide a financial return to the University.
The growing use of complex numerical models across different fields of engineering and the environmental sciences generates a need for GSA tools. These tools enable users to implement a quantitative and structured approach to model calibration and evaluation. In complex nonlinear models unstructured approaches (e.g. trial-and-error, expert judgement) quickly become unfeasible. Establishing rigorous processes for model validation has become a formal requirement for institutions that develop and use their own models, e.g. through "Solvency II" EU Directive in the Insurance and Reinsurance Industry (2009/138/EC, Article 124 "Validation standards"). GSA tools can provide organisations with a rigorous method for meeting these requirements. GSA tools can also analyse uncertainty propagation through models. Quantification of uncertainty in model predictions is increasingly regarded an essential requirement to ensure robust decision-making. Identifying the major sources of uncertainty enables model developers to prioritize efforts for uncertainty reduction. SAFE-com will provide modellers with tools to evaluate their predictions and uncertainties in the context of regulatory analysis and policy appraisal, and in their interactions with policymakers.
The availability of the SAFE toolbox could significantly improve the performance of environmental consultancy, help businesses and government to plan for environmental hazards and manage environmental change, manage uncertainty in the insurance industry, and improve policy decisions based on complex data; all of which are key priorities for NERC.
The outcome of the FoF project would be a customer-focused toolkit, which is fit for purpose and meets the needs of a diverse range of users in different market sectors.
For example, just one of these sectors, environmental consultancy, is a significant global market which continues to grow, with an estimated value of $27.4bn in 2014, and is predicted to expand further over the next five years. The revenues from successful exploitation of the tool within one or more of these markets can be expected to provide a financial return to the University.
The growing use of complex numerical models across different fields of engineering and the environmental sciences generates a need for GSA tools. These tools enable users to implement a quantitative and structured approach to model calibration and evaluation. In complex nonlinear models unstructured approaches (e.g. trial-and-error, expert judgement) quickly become unfeasible. Establishing rigorous processes for model validation has become a formal requirement for institutions that develop and use their own models, e.g. through "Solvency II" EU Directive in the Insurance and Reinsurance Industry (2009/138/EC, Article 124 "Validation standards"). GSA tools can provide organisations with a rigorous method for meeting these requirements. GSA tools can also analyse uncertainty propagation through models. Quantification of uncertainty in model predictions is increasingly regarded an essential requirement to ensure robust decision-making. Identifying the major sources of uncertainty enables model developers to prioritize efforts for uncertainty reduction. SAFE-com will provide modellers with tools to evaluate their predictions and uncertainties in the context of regulatory analysis and policy appraisal, and in their interactions with policymakers.
The availability of the SAFE toolbox could significantly improve the performance of environmental consultancy, help businesses and government to plan for environmental hazards and manage environmental change, manage uncertainty in the insurance industry, and improve policy decisions based on complex data; all of which are key priorities for NERC.
Organisations
| Description | IP Pragmatics have contacted potential end-users in the following sectors: environmental consulting, oil and gas, insurance, transport, engineering, mathematical modelling and economic modelling. Twelve respondents were interviewed from a range of the sectors about the software and sensitivity analysis methods used within their companies, importance of the features of the SAFE toolbox and training and support that they may expect for the toolbox. The responses fell into three categories depending on a range of factors including their similarity in the type and complexity of the models used, the type of industry and requirements of their clients, and their level of knowledge regarding sensitivity analysis: 1. Do not see much need for advanced sensitivity analysis but may be interested in learning more about the individual methods. 2. Understand the need for advanced sensitivity analysis and were interested in how the toolbox could work for them. 3. Aware of the potential need for advanced sensitivity analysis but need further education on the benefits and methods. Numerical examples and real-world case studies from their own industry were seen as important across all three groups, primarily based on the fact that the majority of respondents were not aware of all the tools included in the toolbox. The importance of a graphical user interface and visualisation tools varied depending on the group: Group 1 thought they were not important; Group 3 thought they may were quite useful or important and Group 2 saw them as essential. The feedback regarding a complete suite of tools, a modular structure and a standalone product also followed the same pattern. The importance of access to code showed a slightly different trend with Group 2 thinking it is very important/essential, Group 1 thinking it is important and Group 3 not thinking it is important but may be useful. These particular differences seem to be related to the modeling sophistication of the respondents and their in-house capabilities. The group that was most receptive to the tool was Group 2, so their requirements should be given priority in the next stage of development. |
| Exploitation Route | The key conclusions from the market assessment of the University's SAFE tool to support global sensitivity analysis, which have been outlined in the preceding sections of this report are summarised below. • Curiosity in a range of different industry sectors as to what the SAFE toolbox could provide • Significant uptake in the academic community, despite limited advertising • Several of the techniques in the toolbox are known to be useful by the target sectors • Wide range of potential industry sectors that use mathematical modeling • Product/services concept has not yet been fully identified • Market potential for the toolbox is difficult to quantify at this stage • Uncertainty communication and risk evaluation are of growing interest, with several projects and special interest groups identified in this area • Interest from some interviewees in establishing collaborative partnerships • The ongoing development of the toolbox will provide opportunities to collaborate in a wider range of situations and demonstrate a range of benefits for a range of different industries • A number of different routes to market (inc training, consultancy, spinout or licensing) may be open if commercial proof of concept in a variety of settings can be demonstrated • Potential to partner with multiple software companies and model builders with different routes to market • Regulatory interest may drive wider uptake and use in some sectors • May be difficult to identify the best early adopter sector for the toolbox The main conclusion based on these results was that further case studies need to be created to demonstrate SAFE in settings closely related to industry applications. It was further concluded that collaborations with individual users with larger interest should be pursued initially. |
| Sectors | Aerospace, Defence and Marine,Digital/Communication/Information Technologies (including Software),Energy,Environment,Financial Services, and Management Consultancy,Manufacturing, including Industrial Biotechology |
| Description | The findings clearly demonstrated that a wider range of case study applications of our SAFE toolbox is needed to have a chance of wider uptake by industry. We have since set-up interactions with a range of companies (including JBA, RMS and Airbus) who are now in different stages of testing our SAFE software as suggested by the market research performed as part of this pathfinder project. We have since been awarded two fellowships to work closely with the insurance industry (through NERC) and the water/consultancy industry (through EPSRC). We have further established a joint PhD project with JBA Trust on GSA use for understanding flood model uncertainty. Efforts to include our software in the OASIS CAT modelling platform are underway. |
| Sector | Aerospace, Defence and Marine,Agriculture, Food and Drink,Chemicals,Electronics,Energy,Environment,Healthcare,Manufacturing, including Industrial Biotechology,Other |
| Impact Types | Economic |
| Description | Environmental Change Challenge Fellowships |
| Amount | £723,003 (GBP) |
| Funding ID | EP/R007330/1 |
| Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
| Sector | Public |
| Country | United Kingdom |
| Start | 07/2017 |
| End | 06/2022 |
| Description | Knowledge Exchange Fellowships |
| Amount | £154,260 (GBP) |
| Funding ID | NE/R003734/1 |
| Organisation | Natural Environment Research Council |
| Sector | Public |
| Country | United Kingdom |
| Start | 11/2017 |
| End | 10/2020 |