NSF-EPSRC:ShiRAS. Towards Safe and Reliable Autonomy in Sensor Driven Systems.

Lead Research Organisation: University of Sheffield
Department Name: Automatic Control and Systems Eng

Abstract

Modern data-driven algorithms trained over enormous datasets have revolutionised contemporary autonomous systems with their accurate predictive power. However, due to technical limitations, it is a challenge to integrate large-scale data from many different and complex sensors. Capturing the confidence of these algorithms also remains a challenge.

In response to this demand, ShiRAS will develop pioneering approaches that will introduce autonomy at different levels in sensor-driven systems. The main focus is on machine learning methods with quantified uncertainty of the provided solutions.

Within the field of machine learning, deep learning approaches have resulted in the state-of-the-art accuracy in visual object detection, speech recognition and translation, and many other domains. Deep learning can discover intricate structure in large data sets by using multiple levels of representation, where each level is a higher, more abstract representation of the data. However, a rigorous mathematical framework for uncertainty propagation and update in machine learning models has been largely underexplored. Most current deep learning techniques process the raw data in a deterministic way and do not capture model confidence or trust. Uncertainty can emanate from the noise in the raw data and the parameters of the approach and this impact is a critical part for any predictive system's output.

By representing the unknown parameters using distributions instead of point estimates and propagating these distributions from the input to the output of the system, we propose promising machine learning methods able to handle uncertainty in a unified way.

Planned Impact

The project is multidisciplinary and strongly aligned with the EPSRC priorities for a Prosperous Nation, aiming at informed and connected nation. The project will allow two research teams from the UK and USA to collaborate on a highly interdisciplinary area, together with industrial partners such as QinetiQ, Cisco, Valerann Ltd. and other companies. It will enable them to focus on research and to generate scientific innovations that have the potential to make a difference in key areas with strong demands of autonomy - especially in intelligent transport systems and surveillance. The technologies are aimed to be modular and scalable with respect to the time, space and to large volumes of data. The project will achieve holistic impact in three main areas: (i) Academic, (ii) Societal and (iii) Impact on industrial sectors in accordance with the UK governmental overarching strategies.

As such, ShiRAS will generate considerable impact for a wide range of academic and non-academic beneficiaries, principal amongst whose are:

1) The machine community including academia and industry;
2) Our collaborating industrial partners, directly and other companies;
3) The research community, particularly in the areas of engineering; statistics, computer science
4) The project personnel: the NSF and EPSRC funded postdoctoral researchers
5) The general public, schools and the society in general.

The developed technology can be also beneficial for individual users, the general public, for city councils, policy makers and first responders that need different levels of autonomy, and especially for processing the data provided by multiple sensors.

We will undertake a number of specific activities:
- Visits of the researchers and staff from the UK and USA partners.
- Annual workshops to bring together the consortium and wider users group to exchange information and update users on progress.
- A dedicated web site will be developed to enable access to latest results and progress. Latest news and significant advances will be considered for release to the wider media as appropriate.

Future exploitation: A successful outcome of the project will lead to new projects both via collaboration with our business partners, via Innovate UK, EPSRC/UKRI and other funding sources. Where appropriate we will seek the advice of our respective Research and Enterprise teams to support the uptake of the technology via Industrial, Knowledge Transfer Network and other follow-on funding initiatives. The project will afford new partnerships to be formed based on the EPSRC funding. Since the partners are actively involved in many external organisations and networks we will be able to link strongly with these organisations to strengthen the research base.
Hence, the outcomes of this project will have broad scientific, social and economic impacts. We expect also the team of this project to generate further collaborative contributions based on this project, at national and international level through the US collaboration.
Overall, the project results will shape the future of areas of machine learning and Artificial Intelligence, so that the UK-USA collaboration increases and strengthens the world leadership position in these areas.

Publications

10 25 50

publication icon
Wang P (2021) Feature-refined box particle filtering for autonomous vehicle localisation with OpenStreetMap in Engineering Applications of Artificial Intelligence

publication icon
Liu X (2023) Risk-Aware Contextual Learning for Edge-Assisted Crowdsourced Live Streaming in IEEE Journal on Selected Areas in Communications

publication icon
Buelta A (2021) A Gaussian Process Iterative Learning Control for Aircraft Trajectory Tracking in IEEE Transactions on Aerospace and Electronic Systems

publication icon
Aftab W (2021) A Learning Gaussian Process Approach for Maneuvering Target Tracking and Smoothing in IEEE Transactions on Aerospace and Electronic Systems

publication icon
Cui H (2023) Uncertainty-Aware Variational Inference for Target Tracking in IEEE Transactions on Aerospace and Electronic Systems

 
Description We have developed methods for uncertainty quantification in machine learning methods and autonomous signal processing. We collaborated actively with the Rowan University, USA, Dstl UK, King Juan Carlos University Madrid, Spain and other partners.
When the input data have errors and uncertainties, these impact the final solutions. With the developed approaches we propagate a measure characterising the impact of the data uncertainties and we characterise the trustworthiness of the final solution, depending the level of the noise in the data.

We developed also a Gaussian process approach for extended object tracking and for it studied the impact of the noises on the output results.

We developed also a computationally efficient symmetric diagonally dominant matrix projection-based gaussian process approach. The theoretical results have been validated on several case studies - video surveillance in cities, air quality monitoring and others.
Exploitation Route We have been expanding the current results to surveillance, manufacturing and health systems
Sectors Aerospace, Defence and Marine,Agriculture, Food and Drink,Digital/Communication/Information Technologies (including Software),Environment,Healthcare

URL https://ieeexplore.ieee.org/document/9185002?denied=
 
Description We have extended the project finding to manufacturing and healthcare systems. For manufacturing, we have developed approaches that can measure remotely the size of steel sections in a high temperature steel production process. The process is subject to evaporation, temperature above 1000 degrees, Celsius and illumination changes. The results are both with thermal camera data and Go Pro optical camera data. For Health systems, we have developed machine learning methods for autonomous data processing - for sleep apnoea diagnostics. For video surveillance we validated the theoretical results over real public image and video data. Case studies on air quality monitoring in cities were considered as part of the validation of Gaussian process methods quantifying the impact of sensor errors and other uncertainties of the prediction results.
First Year Of Impact 2020
Sector Aerospace, Defence and Marine,Digital/Communication/Information Technologies (including Software),Environment,Manufacturing, including Industrial Biotechology,Transport
 
Description Contribution to training/educational developments for postgraduates/research users (including courses and course material).
Geographic Reach Multiple continents/international 
Policy Influence Type Influenced training of practitioners or researchers
Impact Machine learning, data science and AI applications are predicted to have an annual economic impact of $9.5 trillion to $15.4 trillion in the next decade; however, a recent McKinsey report anticipates a workforce shortage of ~1.5 M people in data science and AI. The need for workforce development and, in turn, expanding the reach of AI research and education, is therefore critical as well as it is clear. As part of this project, in collaboration with the PI from Rowan University, the PI from Sheffield developed and delivered a new postgraduate course on Machine Vision where the latest developments from the vision methods are included, especially towards resilient multi-sensor data solutions. The project has educational impact since it contributes to strengthening the research capability in key research areas.
 
Description AUTONOMOUS METHOD FOR DETECTING CUTTING TOOL AND MACHINE TOOL ANOMALIES IN MACHINING
Amount £1,033,385 (GBP)
Funding ID EP/T024291/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 08/2020 
End 08/2023
 
Description CSI:Cobot
Amount £450,000 (GBP)
Funding ID https://www.york.ac.uk/assuring-autonomy/projects/csi-cobot/ 
Organisation University of Sheffield 
Sector Academic/University
Country United Kingdom
Start 10/2019 
End 06/2021
 
Description Protecting environments with unmanned aerial vehicle swarms
Amount £7,800,000 (GBP)
Organisation Windracers 
Sector Charity/Non Profit
Country United Kingdom
Start 07/2022 
End 07/2024
 
Description SIGNETS:Signal and information Processing for Scalable Decentralised Intelligent Networks
Amount $1,000,000 (USD)
Funding ID https://www.sheffield.ac.uk/acse/news/signets-advance-fundamental-research-distributed-sensing 
Organisation University of Sheffield 
Sector Academic/University
Country United Kingdom
Start 11/2020 
End 11/2023
 
Description UKRI Trustworthy Autonomous Systems Node in Resilience
Amount £3,033,177 (GBP)
Funding ID EP/V026747/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 11/2020 
End 04/2024
 
Title A computationally efficient Gaussian process approach thanks to the Neumann series 
Description Although kernel approximation methods have been widely applied to mitigate the cost of the kernel matrix inverse in Gaussian process methods, they still face computational challenges. The 'residual' matrix between the covariance and the approximating component is often discarded as it prevents the computational cost reduction. We propose a computationally efficient Gaussian process approach that achieves better computational efficiency, compared with standard Gaussian process methods. Hence, the new approach offers efficient solutions and can be used when dealing with big data sets. 
Type Of Material Technology assay or reagent 
Year Produced 2021 
Provided To Others? Yes  
Impact The approach has been used for air quality estimation and for an application that can be useful to researchers, polocy makers, governmental organisations, e.g. city councils and companies. 
URL https://www.sciencedirect.com/science/article/abs/pii/S0165168421000736
 
Title A method for uncertainty quantification 
Description During 2019-2020, we developed an approach for uncertainty quantification by propagating a distribution. In this way we can characterise the impact of the input data uncertainties over the solutions and outputs of the machine learning approaches. We derived a mean-covariance propagation framework using first-order Taylor series approximation. Then, we extended this first-order approximation to an unscented framework that propagates sigma points through the model layers. The unscented framework is accurate to at least the third-order approximation of the posterior distribution. Instead of working with unscented transforms, we developed also a solution based on quadrature rules which characterises well the impact of data uncertainties on the inputs. We demonstrated these approaches on Convolutional Neural Networks and on Bayesian Convolutional Neural Networks with public data sets CIFAR, MNIST and others. 
Type Of Material Improvements to research infrastructure 
Year Produced 2020 
Provided To Others? Yes  
Impact These results led to a new collaboration with the Liberty Speciality Steels. Our US collaborator established new collaborations with Lockheed Martin Inc. on the application to synthetic-aperture radar images, with the Federal Aviation Administration (FAA) on the application to aviation safety and helipad detection and with the University of Alabama at Birmingham School of Medicine on the application to Brain tumor segmentation (BraTS). 
URL https://ieeexplore.ieee.org/abstract/document/9207214
 
Title Gaussian Process Methods for extended and Group Object Tracking 
Description A Gaussian process method for extended and group object tracking was developed. The method affords both spatial and temporal estimation of the objects of interest, provides an efficient way of fusing of multiple types of sensor data and on-line estimation of the hyperparameters. This is a pioneering work providing on-line learning of the unknown hyperparameters and use for tracking under challenging testing scenarios. The impact of measurement noises over the proposed solutions has also been studied. 
Type Of Material Technology assay or reagent 
Year Produced 2021 
Provided To Others? Yes  
Impact The method led to new developments of spatio-temporal inference. The method led to new research collaborations - now with the University of Surrey and continuing of the collaboration with the University of Cambridge, with a new joint project. 
URL https://ieeexplore.ieee.org/document/9185002
 
Description Gaussian Process Iterative Learning Control for Aircraft 
Organisation King Juan Carlos University
Country Spain 
Sector Academic/University 
PI Contribution We have collaborated by creating a Gaussian process approach for learning control of an aircraft. The results are reported in this IEEE Transactions on Aerospace and Electronic Systems, paper 57(6), 2021
Collaborator Contribution The team from from the Universidad Rey Juan Carlos, Madrid, Spain delivered the mail control aspects of this joint work and validated them.
Impact Buelta A., Olivares A., Staffetti E., Aftab W., and Mihaylova L., (2021), A Gaussian Process Iterative Learning Control for Aircraft Trajectory Tracking, IEEE Transactions on Aerospace and Electronic Systems, 57(6): 3962-3973.
 
Description Gaussian Process Iterative Learning Control for Aircraft 
Organisation King Juan Carlos University
Country Spain 
Sector Academic/University 
PI Contribution We have collaborated by creating a Gaussian process approach for learning control of an aircraft. The results are reported in this IEEE Transactions on Aerospace and Electronic Systems, paper 57(6), 2021
Collaborator Contribution The team from from the Universidad Rey Juan Carlos, Madrid, Spain delivered the mail control aspects of this joint work and validated them.
Impact Buelta A., Olivares A., Staffetti E., Aftab W., and Mihaylova L., (2021), A Gaussian Process Iterative Learning Control for Aircraft Trajectory Tracking, IEEE Transactions on Aerospace and Electronic Systems, 57(6): 3962-3973.
 
Description The University of Surrey 
Organisation University of Surrey
Country United Kingdom 
Sector Academic/University 
PI Contribution We have started a collaboration in the area of distributed information networks.
Collaborator Contribution The project called SIGNeTs is led by the University of Cambridge, with partners the University of Sheffield and the University of Surrey. The SIGNeTs project can be seen as a continuation of SHiRAS. SIGNeTs is funded by Dstl and the USA MoD.
Impact We are starting the collaboration with Surrey.
Start Year 2020
 
Title A Deep Learning Framework for Autonomous Flame Detection 
Description The MATLAB codes for the paper 'A Deep Learning Framework for Autonomous Flame Detection'. 
Type Of Technology Software 
Year Produced 2021 
Open Source License? Yes  
Impact The code can be used for reproducible research. 
URL https://figshare.shef.ac.uk/articles/software/A_Deep_Learning_Framework_for_Autonomous_Flame_Detecti...
 
Title MATLAB Codes for Remote Steel Section Sizing with GoPro Camera 
Description Wang, Peng (2021): MATLAB Codes for Remote Steel Section Sizing with GoPro Camera. The University of Sheffield. Composition. https://doi.org/10.15131/shef.data.13705654.v1 
Type Of Technology Software 
Year Produced 2020 
Open Source License? Yes  
Impact This is the code written as part of the collaboration with the Liberty Speciality Steels. The solutions for remote steel sizing include quantification of uncertainties. 
URL https://figshare.shef.ac.uk/articles/composition/MATLAB_Codes_for_Remote_Steel_Section_Sizing_with_G...
 
Title MATLAB Codes for Remote Steel Section Sizing with Thermal Camera 
Description This is a software for remote steel section sizing. It is linked with this publication: Wang, Peng; Lin, Yueda; Mihaylova, Lyudmila (2021): MATLAB Codes for Remote Steel Section Sizing with Thermal Camera. The University of Sheffield. Composition. https://doi.org/10.15131/shef.data.13713598.v1 
Type Of Technology Software 
Year Produced 2021 
Open Source License? Yes  
Impact The sensing and autonomous image data processing has been done in collaboration with the Liberty Speciality Steels. This work opens new doors for the company to automate some of the stages of steel production. 
URL https://doi.org/10.15131/shef.data.13713598.v1
 
Title Software for the work pn Bayesian Neural Networks Uncertainty Quantification with Cubature Rules 
Description Software for this work: P. Wang, R. He, Q. Zhang, J. Wang, L. Mihaylova, N. Bouaynaya, Bayesian Neural Networks Uncertainty Quantification with Cubature Rules, In Proc. of the International Joint Conference on Neural Networks (IJCNN), 2020, Glasgow, Scotland, UK, July 2020, DOI: 10.1109/IJCNN48605.2020.9207214 
Type Of Technology Software 
Year Produced 2020 
Open Source License? Yes  
Impact Contribution to uncertainty quantification in Bayesian networks 
URL https://github.com/grapesonwang/BayesianNN_Uncertainty
 
Description A talk on: "UKRI Uncertainty-Aware Machine Learning Methods for Trustworthy Autonomous Systems" 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact A talk at the "Trusting Machines? Cross-sector Lessons from Healthcare and Security" event The event was created as part of the collaboration between RUSI, the TAS Hub and National Gallery X.

The topic of the toc is summarised briefly below:
Sensors provide enormous amounts of information beyond the capacity that a human could process. Being part of autonomous systems, the data are coming in real time and need to be processed quickly and fed to the control and decision-making levels.
This talk will discuss recently developed machine learning methods able to deal with data challenges such as volume, velocity, veracity and variety. The recent trends in machine learning and autonomy are towards development of trustworthy solutions, able to work under different conditions - due to the external environmental changes and the dynamics of the autonomous systems. Autonomous systems need to be safe and reliable, and include both information from hard and soft sensors - cameras, LiDARs, radars, wireless sensor networks and data from the Internet or other sources.
Generic principles valid for at least four domains will be discussed - for surveillance, health, manufacturing and transport systems.
This talk aims to stimulate discussions from multi-disciplinary areas and consider questions like these:
• How could be define different levels of trustworthiness and resilience, respectively?
• How could we characterise trustworthiness? What criteria could we have? How could we quantify the impact of uncertainties and provide resilience?
• What are going to be the next generations of methods for uncertainty-aware autonomous systems?
• How could we fuse reliably the data from multiple heterogeneous sensors, in order to provide resilience?
• How do we link the technological aspects with ethics, societal and human-centred factors?
Year(s) Of Engagement Activity 2021
URL https://rusi.cplus.live/event/trusting-machines-21
 
Description Invited talk at the Harbin Intelligent Navigation and Advanced Information Technology Workshop 2020, Harbin Engineering University, China 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact An invited talk
Talk Title: Machine Learning and Multi-Sensor Data Processing for Autonomous Systems
More than 55 percent of the world population lives in cities and it is predicted that this number will increase to 70 percent in the next five years. This will increase the number of mega cities around the world. Cities are complex systems and they are equipped with many sensors. Sensors provide enormous amounts of information beyond the capacity that a human could process. This talk will focus on the challenges that modern cities face - from the point of view of mobility, intelligent transport, surveillance, resilience, machine learning methods and autonomy in urban data analytics and big data. This talk will discuss recently developed machine learning methods able to deal with data challenges such as volume, velocity, veracity and variety. Some of these trends in machine learning and autonomy are towards development of trustworthy solutions, able to work under different conditions.
Year(s) Of Engagement Activity 2020
URL https://mp.weixin.qq.com/s/Nuor0XaETThgoqBh2HCtxA
 
Description Invited talk on Transition to a More Resilient World - International Virtual Summit 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact A talk at the KES International COVID-19 Challenge: A Transition to a More Resilient World - International Virtual Summit, 29 March 2021
Year(s) Of Engagement Activity 2021
URL http://covidchallenge.kesinternational.org/index.php
 
Description Machine Learning Methods for Autonomous Image and Video Analytics - Invited talk 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact I gave a talk on recent developments on trustworthy machine learning methods for image and video analytics.
Year(s) Of Engagement Activity 2021
URL http://www.icdip.org/
 
Description Towards Trustworthy Machine Learning Methods 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact This is an invited plenary talk as part of the 12th International Conference on Electronics, Communications and Networks (CECNet 2022), November 4th-7th, 2022, Online Conference, Beijing. The talk stimulated interesting discussions and possibly new research collaborations.
Year(s) Of Engagement Activity 2022
URL http://cecnetconf.org/Program
 
Description Towards Trustworthy Machine Learning Methods 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Other audiences
Results and Impact This is an invited talk as part of the Terrorism Risk Assessment, Modelling and Mitigation Seminar Series (TRAMMSS) at Cranfield University, 11 November 2022
Year(s) Of Engagement Activity 2022
URL https://www.cranfield.ac.uk/events/events-2022/towards-trustworthy-machine-learning-methods