Advanced Uncertainty Quantification Techniques for Maintenance Planning

Lead Research Organisation: CRANFIELD UNIVERSITY
Department Name: Sch of Aerospace, Transport & Manufact

Abstract

Modern complex engineering systems (CES) are expected to function effectively whilst maintaining reliability in service. This presents significant challenges to confidently and accurately predict maintenance costs and asset availability. These challenges raise varying degrees of uncertainty stemming from multiple heuristic and statistical sources throughout the in-service life of CES. Techniques to quantify uncertainty from statistical sources are well documented, but methods to obtain and analyse heuristic attributes often go undefined and unmitigated, which raise further uncertainties.

A holistic view is necessary to improve decision-making capabilities and reduce maintenance costs and turnaround time. This project aims to develop an intelligent uncertainty quantification system that learns from a combination of historic equipment data and heuristic estimates to allow the user to forecast the level of uncertainty through the in-service phase of CES.

This work builds on projects undertaken in conjunction with the Through-Life Engineering Services (TES) centre at Cranfield University that have been applied in industry to tackle cost uncertainty at the bidding stage, which face broadly similar challenges as those in service. Core factors that influence uncertainty and hinder confident forecasting include quality of available data, experience and knowledge. These have been examined through collaboration with experts from BAE Systems, along with current practice in uncertainty assessment and future maritime support programmes to address challenges. Accurate forecasts in service depend on reliable data and predictable maintainer performance levels. A holistic view ultimately allows for more accomplished decision-making but requires trade-offs between quality and cost over the asset's life cycle.

Studentship Projects

Project Reference Relationship Related To Start End Student Name
EP/N509450/1 30/09/2016 29/09/2021
1944319 Studentship EP/N509450/1 30/09/2017 24/03/2021
 
Description Current practice and challenges towards handling uncertainty for effective outcomes in maintenance:
Through a series of questionnaires and semi-structured interviews, 6 core challenges were identified that influence uncertainty in industrial maintenance: maintainer performance, quality of information, stakeholder communication, intellectual property rights, resistance to change and technology integration. From the interviews, it was deemed that a holistic view of quantitative and qualitative attributes ultimately allows for more accomplished decision-making but requires trade-offs between quality and cost over the asset's life cycle.

Ranking and selection of approaches identified from literature:
Key approaches identified in literature to enable the aggregation of quantitative and qualitative uncertainty were ranked using TOPSIS. The best suited approaches were amalgamated to optimise pedigree through a coefficient of variation. Uncertainty forecasts increasingly use intelligent learning techniques, though methods applied for the in-service phase are limited. Key challenges faced here, as in traditional uncertainty analysis, are quality of available data and experience and knowledge of the data collector. This limits the ability to optimally train neural networks through probabilistic Bayesian learning, as well as time for maintenance policies to stabilise. This reduces confidence in associated uncertainty estimates from stochastic modelling, leading to over or under estimation. The TOPSIS method was also applied to identify the best suited approaches to forecast uncertainty through the in-service life and form part of the overall framework.

MCDUQ framework:
A multistep compound dynamic uncertainty quantification (MCDUQ) framework has been developed to quantify, aggregate and forecast uncertainty from quantitative and qualitative sources. This is based on existing methodologies identified in literature and over 30 hours of interviews within the defence sector. This mixed modelling approach will use multistep and single step prediction to optimise uncertainty forecasts. The combination of these features to aid decision making in industrial maintenance makes a key intellectual contribution to optimise uncertainty management capabilities for real-life industrial applications.
Exploitation Route Core factors that influence uncertainty originating from challenges in maintenance have been identified, ranked and verified with industrial practitioners. Simulation of these factors in the maintenance model will highlight the level of uncertainty they each contribute to the total system estimate and their impact on maintenance efficiency in context. The developed model can be utilised by industrial maintenance planners to illustrate contrasting predictions of individual and aggregated uncertainty present and forecast for the in-service phase to optimise decision-making and reduce over or under estimation.
Sectors Aerospace

Defence and Marine

Digital/Communication/Information Technologies (including Software)

Manufacturing

including Industrial Biotechology
 
Description Not used yet as award is not complete. Resulting framework will provide real time and predicted levels of uncertainty in maintenance through the in-service life of complex engineering systems. This will provide an individual and holistic view of uncertainties relating to equipment availability and maintenance capability, allowing for more accomplished decision-making. Live and continuous forecasts are beneficial to industry in a number of areas including maintenance planning, digital twins, big data, and lifing studies in the face of exponential increases in technological complexity. Real-world data is required to truly test implementation - not yet able to get hold of
 
Title Data relating to: "An uncertainty quantification and aggregation framework for system performance assessment in industrial maintenance" (2020) 
Description Excel file corresponding to data in conference paper - tables summarising variables used in the paper, calculated in MATLABImages: Figures 1-4 as in conference paperPowerPoint presentationPublished paper 
Type Of Material Database/Collection of data 
Year Produced 2021 
Provided To Others? Yes  
Impact Data contributing to the published paper, enhancing outcomes of the project 
URL https://cord.cranfield.ac.uk/articles/conference_contribution/Data_relating_to_An_uncertainty_quanti...
 
Title Data relating to: "Current practice and challenges towards handling uncertainty for effective outcomes in maintenance" (2019) 
Description Excel file corresponding to data in conference paper:'Details' tab denotes participant experience and pedigree scores'Influencing factors' tab displays questionnaire results and analysis'Influencing factors w. pedigree' looks at how pedigree could be applied directly to questionnaire answers'Pairwise & AHP' shows construction and results of AHP processPowerPoint file: Embedded conference video presentation, summary of paper, comparison of approaches 
Type Of Material Database/Collection of data 
Year Produced 2020 
Provided To Others? Yes  
Impact Data relating to the published paper, enhancing research outcomes 
URL https://cord.cranfield.ac.uk/articles/Current_practice_and_challenges_towards_handling_uncertainty_f...
 
Title Data relating to: "Dynamic multistep uncertainty prediction in spatial geometry" (2020) 
Description Excel file corresponding to training data and results in conference paper - applied in MATLABImages: Figures 1-4 as in conference paperVideo: 3D plot rotationVideo: Conference presentation 
Type Of Material Database/Collection of data 
Year Produced 2021 
Provided To Others? Yes  
Impact Research contributing to the published paper, enhancing outcomes of the project 
URL https://cord.cranfield.ac.uk/articles/conference_contribution/Data_relating_to_Dynamic_multistep_unc...
 
Title Data relating to: "Identifying challenges in quantifying uncertainty: case study in infrared thermography" (2018) 
Description Excel file corresponding to data in conference paper: 'Paper tables' tab contains summary of variables used in the paper, calculated using MATLAB 'Conditions' tab contains recorded temperatures and humidity for each run read by MATLAB 'Readings' tab collates reading values for each run read by MATLAB 'Run1-10' tabs contain data recorded for each run including ROI size and location PowerPoint file: Conference presentation 
Type Of Material Database/Collection of data 
Year Produced 2020 
Provided To Others? Yes  
Impact Data corresponding to the published paper, enhancing research outcomes 
URL https://cord.cranfield.ac.uk/articles/conference_contribution/Identifying_challenges_in_quantifying_...
 
Title Literature database 
Description Excel database detailing publication details for systematic literature review, as well as iterative word counting process for literature synthesis 
Type Of Material Database/Collection of data 
Year Produced 2019 
Provided To Others? No  
Impact Easy to use and access literature analysis 
 
Description BAE Systems summary presentation 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Summary presentation with the industrial funder (BAE Systems) to present the work and explore how they could use the final model and what improvements could be made
Year(s) Of Engagement Activity 2021
 
Description ISMOR workshop 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Industry/Business
Results and Impact A workshop was held as part of the International symposium on military operational research (ISMOR). The purpose was to identify and evaluate applications for machine learning in risk and uncertainty quantification. As 1 of 3 presenters, I gave an overview of uncertainty, methods to assess it, key challenges in doing so and benefits and applications of using machine learning in this context.
Live voting software was used in the presentation along with discussions to understand what approaches are used across industrial sectors and if or how machine learning is implemented. Factors and challenges identified in previous interviews and developed through a previous workshop were validated though the audience's feedback. Due to the popularity of the workshop, 2 sessions were held throughout the afternoon.
Year(s) Of Engagement Activity 2019
 
Description SCAF workshop - Modelling risk and uncertainty 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Hosted by the society for cost analysis and forecasting (SCAF), the workshop focused on the need to assess and simulate risk and uncertainty to inform evidence-based decisions. I gave a presentation entitled "It works most of the time: Assessing the impact of multiple types of uncertainty on decision-making practices for industrial maintenance". This focused on the question of whether a consideration of uncertainties from a mix of statistical data and qualitative factors raise more challenges than individual types on the road to good decision-making.
An assessment was made on the impact of uncertainty influenced by key industrial factors identified in separate interviews with experienced maintenance managers. A live survey was held with the audience to validate the results to establish core factors influencing uncertainty in maintenance. This also sparked a helpful discussion on different approaches industries use to estimate uncertainty and the level of detail considered.
Year(s) Of Engagement Activity 2018
URL http://www.scaf.org.uk/library/prespaper/2018_11/It%20works%20most%20of%20the%20time%20-%20Alex%20Gr...
 
Description SCAF workshop - practical examples of cost estimating & analysis 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Industry/Business
Results and Impact Hosted by the society for cost analysis and forecasting (SCAF), participants presented cost estimation methodologies applied to case studies in partnership with professional teams from academia, industry and consultants. Feedback was given by senior practitioners.
Observation of the presentations, constructive feedback from the review panel and networking with participants revealed best practice techniques applied by cost estimators to assess correlations, analyse uncertainty and aid decision making.
Year(s) Of Engagement Activity 2018
URL http://www.scaf.org.uk/library/prespaper/2018_04/Introduction_Slides.pdf
 
Description TESI lunchtime seminar 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Postgraduate students
Results and Impact An overview of the research project, aims and findings was given in a lunchtime seminar with the Through-Life Engineering Services Institute (TESI) at Cranfield University. Participants included fellow researchers and academics who gave constructive feedback on methodologies and presentation techniques. In addition, possible collaborations were discussed.
Year(s) Of Engagement Activity 2019
 
Description Third Uncertainty Quantification & Management Study Group with Industry 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Around 60 practitioners from a range of academic and industrial backgrounds attended the study group to address uncertainty-related problems presented by 4 industries.
In my group of 13 members addressing 1 of these industries, 3 tasks were discussed: numerical simulation of a surrogate model, validation experiments and sensitivity analysis. The core discussion focused on validation of the model following identification of the most important measurement(s) in the model.
Collaboration with attendees from a range of fields in both academia and industry prompted input from diverse perspectives, allowing for fruitful application of techniques that fit in an industrial context, able to work towards alleviating the problems presented. This helped broaden my view and knowledge on existing challenges and techniques present in the field of uncertainty quantification.
Year(s) Of Engagement Activity 2017
URL https://warwick.ac.uk/fac/sci/wcpm/studygroup3/