Robust remote sensing for multi-modal characterisation in nuclear and other extreme environments

Lead Research Organisation: University of Birmingham
Department Name: Mechanical Engineering

Abstract

This project addresses the problem of "characterisation" of Extreme Environments (EE), by deploying and combining information from a variety of different Remote Sensing modalities. Our principle application area is nuclear decommissioning, however our research outputs will be relevant to other EE.

Before nuclear decommissioning interventions can happen, the facility/plant being decommissioned must be "characterised", to understand: physical layout and 3D geometry; structural integrity; contents including particular objects of interest (e.g. fuel rod debris). 3D plant models must further be annotated with additional sensed data: thermal information; types/levels/locations of contamination (radiological, chemical etc.). Characterisation may be needed before, during or after POCO (Post Operation Clean Out). "Quiescent buildings" may be over half a century old, with uncertain internal layout and contents.

Characterisation is needed in dry environments (e.g. contaminated concrete "caves") and wet environments (e.g. legacy storage ponds). Caves may be unlit, causing difficult vision problems (shadows, contrast, saturation) with robot-mounted spotlights. Underwater environments cause significant visibility degradation for RGB cameras, and render most depth/range sensors unusable. New technologies, e.g. acoustic cameras, engender interesting new challenges in developing algorithms to process these new kinds of image data.

In many cases, robots are needed to deploy Remote Sensors into Extreme Environments and move them to desired locations and viewing poses. In some cases, robots must also assist characterisation by retrieving samples of contaminated materials. In many case real-time Remote Sensing data must also be applied to inform and control the actions of robots, while performing remote intervention tasks in EE.

This project brings together a unique, cross-disciplinary and international team of researchers and institutes, spanning three continents, to address these challenges. End-users NNL and JAEA will advise on scenarios and challenges for Remote Sensing in nuclear environments. Active facilities at JPL will be used to measure degradation of sensors, chips and software under a variety of radiation types and doses. JPL and Essex researchers will use this data to develop new models for predicting such degradation. Essex researchers will then develop new methods for software and embedded hardware design, which overcome radiation damage by incorporating new approaches to fault detection, tolerance and recovery.

The scenarios provided by the partners, and the degradation data measured by JPL, will be used to develop new benchmark data-sets comprising data from multiple sensing modalities (RGB cameras, depth/range cameras, IR thermal imaging, underwater acoustic imaging), featuring a vairiety of nuclear scenes and objects.

UoB and Essex researchers will develop new algorithms for real-time 3D characterisation of scenes, with intelligent and adaptive fusion of multiple sensing modalities. First, new multi-sensor fusion methods will be developed for 3D modelling, semantic/meta-data labelling, recognition and understanding of scenes and objects. Second, these methods will be extended to incorporate new algorithms for overcoming extreme noise and other kinds of degradation in images and sensor data. Third, we will develop the robots and robot control methods needed to: i) deploy remote sensors into extreme environments; ii) exploit remote sensor data to guide robotic interventions and actions in these environments.

Finally, we will carry out experimental deployments of these new technologies. Robust hardware and software solutions, developed by Essex, will be tested in active radiation environments at JPL. We will also carry out experimental robotic deployments of sensor payloads into inactive but plant-representative nuclear environments at NNL Workington and the Naraha Fukushima mock-up testing facilities in Japan.

Planned Impact

This project directly addresses the most important and difficult RAS and Remote Sensing challengs that must be overcome to enable safe, successful and timely decommissioning of difficult legacy nuclear facilities, many of them designated as representing "intolerable risk" to the nation.

Decommissioning and disposal of the 4.9million tons of nuclear waste in the UK, represents the largest environmental remediation project in the whole of Europe, and is projected to cost as much as £220billion. Much of this work can only be done by remote methods, because the high levels of radioactive material are hazardous to humans. This directly engenders fundamental challenges for Remote Sensing in Extreme Environments.

This project will therefore deliver a number of key impacts:
1) It addresses the major UK societal challenge of cleaning up intolerable domestic legacy waste sites (impacting the UK population as a whole as well as future generations).
2) It enhances UK capabilities in decommissioning, where we have internationally recognised expertise, and which open up a >$300billion worldwide market to the UK economy (impacting the nuclear workforce in particular, and the overall UK economy more generally).
3) RAS (Robotics and Autonomous Systems has been identified as "one of the 'eight great technologies' which will propel the UK to future growth". This project will directly develop the Remote Sensing methods and Extreme Environment hardware designs that are needed to enable UK RAS expertise to penetrate the economically large, and societally important, market of nuclear decommissioning and remediation.
4) It is a very strong fit to the existing EPSRC portfolio, including:
i) The EPSRC DISTINCTIVE (Decommissioning, Immobilisation and Storage soluTIons for NuClear wasTe InVEntories) consortium.
ii) The EPSRC AIS (Autonomous and Intelligent Systems) program, in which the UK nuclear industry is a major stakeholder.
iii) The major characterisation and robotics component of te EPSRC UK-Korea Nuclear Program.
iv) The remote sensing, characterisation and robotics components of the EPSRC UK-Japan calls.
5) We have proposed specific pathways to impact, through:
i) the successful embedded systems start-ups of C-I McDonald-Maier;
ii) our Knowledge Transfer collaboration with the UK subsidiary of a major global industrial robotics company;
iii) our direct collaborations with nuclear end-users in UK, Europe, Korea and Japan.
Much of the new technology developed during the project can find direct route to market through these collaborations. Thus, the proposed research will directly benefit UK industry, as well as enhancing the nuclear and robotics capabilities available to end-users worldwide.
6) We have proposed significant educational outreach and public communication of science activities to run in parallel with the research, thereby directly engaging with the non-expert population, promoting public awareness of the nuclear and other EE challenges, and promoting study and career interest from young people in Remote Sensing specifically, and science and engineering more generally.
 
Description We have created a novel dataset of nuclear waste-like objects (rubber gloves, swabs, respirators, metal, chain, hoses, cans, bottles, etc). 3D models have been made of all objects. We have developed a "virtual camera" method for automatically generating virtual images of each object from multiple views, and this has then been used as training data fior machine learning. We have developed new methods, based on deep learning neural networks, for real-time 3D reconstruction of scenes, detecting, recognising, segmenting and 3D modeling of uclear waste objects. We have also demonstrated learning-based recognirion of materials (e.g. concrete, metal, cermamic, fabric, painted surface, glass etc). These capabilities are fundamental for "characterisation" in nuclear decommissioning.

Our collaboration with NASA Jet propulsion Lab has helped us to understand that the space industry is significantly more advanced than the nuclear industry in their approaches to radiation-resilient embedded systems. In particular, the space industry has pioneered use of Radiation Hardening by Software (RHBSW) in which a small percentage of CPU effort is dedicated to checking, verification, error detection and correction. Previously, nuclear industry focused on physical approaches and shielding - this means that rad-hardened chips are typically very expensive and also the latest COTS processors (much more powerful) are not available in a rad-hardened form. Therefore RHBSW offers the potentia to use e.g. the latest GPU processors for advanced computer vision, while also cheaply and conveniently making them more resilient and robust via software approaches.

During 2018, further development work has been undertaken on our nuclear waste objects dataset, and Essex have undertaken new experiments to expose processors and sensors (e.g. microsoft kinect RGBD camera) to varying doses of radiation, utilising collaborative support from Rutherford Appleton Lab.
Exploitation Route See above - significant potential to port RHBSW from space industry to nuclear industry.
Sectors Aerospace, Defence and Marine,Digital/Communication/Information Technologies (including Software),Education,Electronics,Energy,Environment,Healthcare,Manufacturing, including Industrial Biotechology,Security and Diplomacy,Other

 
Description In addition to the intended nuclear industry end-users, this project has created a strong collaboration with NASA Jet Propuslsion Lab, as well as with several SMEs and spinouts. This led to a new collaboration with InnovateUK SBRI funding to demonstrate proof of principle of a vision-guided semi-autonomous nuclear decommissioning robot. Also, this collaboration with Essex and NASA contributed very strongly to the creation of the new £42million National Centre for Nuclear Robotics initiative. The collaboration is showing how advanced radiation resilience approaches can be transferred from the Space domain to the nuclear industry domain. The collaboration is also developing new computer vision methods and showing how the latest advances in deep learning and other methods from academia can have practical application in a highly conservative and risk-averse industry. In 2018, we have been contacted by SME startup Hybird who work on unmanned air vehicles for inspection. This has led to tech transfer work by Ehsan (Essex) and a new collaborative InnovateUK proposal.
First Year Of Impact 2017
Sector Aerospace, Defence and Marine,Education,Electronics,Energy,Environment,Security and Diplomacy,Other
Impact Types Societal,Economic,Policy & public services

 
Description National Centre for Nuclear Robotics RAI Hub
Amount £11,000,000 (GBP)
Funding ID EP/R02572X/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Academic/University
Country United Kingdom
Start 10/2017 
End 03/2022
 
Description InnovateUK SBRI collaboration - nuclear decommissioning robot 
Organisation National Nuclear Laboratory
Country United Kingdom 
Sector Public 
PI Contribution University of Birmingham researchers, via our spinout consultancy A.R.M Robotics Ltd (a specialised robotics SME), collaborated with Bristol Robotics Lab, National Nuclear Lab, and several other SMEs, as part of an InnovateUK SBRI project. In this project we built and demonstrated a semi-autonomous mobile manipulator robot, which could autonomously navigate and cut pipework. The contribution of the Birmingham team was the advanced vision system and robot arm control.
Collaborator Contribution Bristol Robotics Lab assembled the robot platform, provided the semi-autonomous vehicle navigation software, and hosted the project demo. NNL Ltd provided project management and expert nuclear industry advice and guidance. University of Bristol provided expert advice on radiometric sensor payload for the robot. University of Essex provided expert advice about potential for making the onboard computer systems radiation resilient.
Impact Highly successful feasibility study. Delivered working prototype and successful live demonstration of autonomously navigating across a room, building up a 3D model of pipework, and autonomously cutting the pipe with a tool on a vehicle-mounted robot arm.
Start Year 2017
 
Description InnovateUK SBRI collaboration - nuclear decommissioning robot 
Organisation University of Bristol
Country United Kingdom 
Sector Academic/University 
PI Contribution University of Birmingham researchers, via our spinout consultancy A.R.M Robotics Ltd (a specialised robotics SME), collaborated with Bristol Robotics Lab, National Nuclear Lab, and several other SMEs, as part of an InnovateUK SBRI project. In this project we built and demonstrated a semi-autonomous mobile manipulator robot, which could autonomously navigate and cut pipework. The contribution of the Birmingham team was the advanced vision system and robot arm control.
Collaborator Contribution Bristol Robotics Lab assembled the robot platform, provided the semi-autonomous vehicle navigation software, and hosted the project demo. NNL Ltd provided project management and expert nuclear industry advice and guidance. University of Bristol provided expert advice on radiometric sensor payload for the robot. University of Essex provided expert advice about potential for making the onboard computer systems radiation resilient.
Impact Highly successful feasibility study. Delivered working prototype and successful live demonstration of autonomously navigating across a room, building up a 3D model of pipework, and autonomously cutting the pipe with a tool on a vehicle-mounted robot arm.
Start Year 2017
 
Description InnovateUK SBRI collaboration - nuclear decommissioning robot 
Organisation University of Essex
Country United Kingdom 
Sector Academic/University 
PI Contribution University of Birmingham researchers, via our spinout consultancy A.R.M Robotics Ltd (a specialised robotics SME), collaborated with Bristol Robotics Lab, National Nuclear Lab, and several other SMEs, as part of an InnovateUK SBRI project. In this project we built and demonstrated a semi-autonomous mobile manipulator robot, which could autonomously navigate and cut pipework. The contribution of the Birmingham team was the advanced vision system and robot arm control.
Collaborator Contribution Bristol Robotics Lab assembled the robot platform, provided the semi-autonomous vehicle navigation software, and hosted the project demo. NNL Ltd provided project management and expert nuclear industry advice and guidance. University of Bristol provided expert advice on radiometric sensor payload for the robot. University of Essex provided expert advice about potential for making the onboard computer systems radiation resilient.
Impact Highly successful feasibility study. Delivered working prototype and successful live demonstration of autonomously navigating across a room, building up a 3D model of pipework, and autonomously cutting the pipe with a tool on a vehicle-mounted robot arm.
Start Year 2017
 
Description InnovateUK SBRI collaboration - nuclear decommissioning robot 
Organisation University of the West of England
Country United Kingdom 
Sector Academic/University 
PI Contribution University of Birmingham researchers, via our spinout consultancy A.R.M Robotics Ltd (a specialised robotics SME), collaborated with Bristol Robotics Lab, National Nuclear Lab, and several other SMEs, as part of an InnovateUK SBRI project. In this project we built and demonstrated a semi-autonomous mobile manipulator robot, which could autonomously navigate and cut pipework. The contribution of the Birmingham team was the advanced vision system and robot arm control.
Collaborator Contribution Bristol Robotics Lab assembled the robot platform, provided the semi-autonomous vehicle navigation software, and hosted the project demo. NNL Ltd provided project management and expert nuclear industry advice and guidance. University of Bristol provided expert advice on radiometric sensor payload for the robot. University of Essex provided expert advice about potential for making the onboard computer systems radiation resilient.
Impact Highly successful feasibility study. Delivered working prototype and successful live demonstration of autonomously navigating across a room, building up a 3D model of pipework, and autonomously cutting the pipe with a tool on a vehicle-mounted robot arm.
Start Year 2017
 
Description National Centre for Nuclear Robotics 
Organisation University of Essex
Country United Kingdom 
Sector Academic/University 
PI Contribution U Birmingham (Stolkin) is the leader of this collaboration consortium.
Collaborator Contribution U Essex (Sensing in Extreme Environments grant) became a key member of NCNR and also brought NASA Jet Propulsion Lab to NCNR as a collaborator. Work done in UK-Korea Civil Nuclear Collaboration also contributed to us later winning NCNR, and brought Korean Atomic Energy Research Agency to NCNR as a collaborator.
Impact Engagement with large number of UK universities, very large industrial partnership, international collaborations, and parallel commercialisation projects with InnovateUK and SME companies.
Start Year 2017
 
Description One week robotics/AI summer school on nculear robotics for 14-17 yr olds at Royal Institution 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact This is a one week course, run by Prof Stolkin and Peter Brewer at Royal Institution of Great Britain, in their prestigious London |Mayfair premises. 24 school children, from 14-17, spend one week working in teams to design, build and prorgam their own robots, inspired by the challenges of nuclear decommissioning. This includes underwater robots, and autonomous robots that use sensors to navigate and manipulate materials in a mockup nuclear environment. The students also learn about otehr areas of robotics, vision systems, AI and machine learning from a series of mini-lectures, as well as how robotics relates to areas of mathematics and physics. The students also learn about Intellectual Property, patenting and entrepreneurialism, and draft their own patent claims for the robots that they develop. We have received outstanding feedback on this coures, and are working with Ri to make these materials scalable for use nationally and internationally.
Year(s) Of Engagement Activity 2018
 
Description Presentation to IEEE Technical Committee on Robotics and Automation for Nuclear Facilities (RANUF) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Invited talk for IEEE Tech,. Committee RANUF forum on robotics for nuclear applications.
Year(s) Of Engagement Activity 2017
 
Description Presentation to IEEE Technical Committee on Robotics and Automation for Nuclear Facilities (RANUF) at IROS 2017 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Invited talk at the nuclear forum at IROS international robotics conference. This forum/workshop created by IEEE Tech Committee on Robotics and Automation for Nuclear Facilities.
Year(s) Of Engagement Activity 2017
 
Description Presented NCNR to the Royal Society 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Industry/Business
Results and Impact Keynote talk at Royal Society event aimed at Industry Fellows and entreprreneurship. Held at Warwick Manufacturing Centre
Year(s) Of Engagement Activity 2018
 
Description Visit of MP. Rt. Hon. Thangam Debbonaire - Parliamentary Fellow for Energy Industry) - report to parliament 
Form Of Engagement Activity Participation in an open day or visit at my research institution
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Policymakers/politicians
Results and Impact House of Commons MP, Thangam Debbonaire, visited the Extreme Robotics Lab (PI Stolkin) at University of Birmingham. Stolkin gave a presentation on National Centre for Nuclear Robotics, and the ERL team then gave a live demo of advanced vision-guided autonomous robotic grasping and manipulation.

Debbonaire is the Industry and Parliament Trust (IPT) Fellow for the Energy Industry. This gives her a remit to investigate the energy industry and report back to parliament. Ms. Debbonaire was very impressed by our robotics work, and interested in how robotics could link to the energy sector. She said that she will be mentioning this work in her report to parliament. Ms. Debbonaire also expressed great interest in our educational outreach work, and intends to follow up on this with us for further discussions.
Year(s) Of Engagement Activity 2018
 
Description Won/led Nuclear workshop at European Robotics Forum 2017 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact We were competitively awarded (peer review) a prestigious workshop at European Robotics Forum. This attracted international speakers and a large audience. It is very difficult to win such workshops at ERF. This was first time that nuclear has been a workshop theme, and we were then able to win a nuclear workshop successively in 2017 and 2018.
Year(s) Of Engagement Activity 2017
 
Description Won/led Nuclear workshop at European Robotics Forum 2018 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact We were competitively awarded (peer review) a prestigious workshop at European Robotics Forum. This attracted international speakers and a large audience. It is very difficult to win such workshops at ERF. This was first time that nuclear has been a workshop theme, and we were then able to win a nuclear workshop successively in 2017 and 2018.
Year(s) Of Engagement Activity 2018