Intelligent Workspace Acquisition, Comprehension and Exploitation for Mobile Autonomy in Infrastructure Denied Environments

Lead Research Organisation: University of Oxford
Department Name: Engineering Science

Abstract

Vehicles will only get smarter. There will always be a desire for more machine intelligence and autonomy. Our needs and expectations are ever increasing. As a result, we continue to pack more sensors and more computation into the robots that carry, transport, labour for and defend us.

Here we interpret autonomy as a robot's ability to sense, understand and ultimately act of its own accord in its operating environment. This proposal is about giving autonomous vehicles the ability to navigate in difficult conditions over long periods of time. Conditions become "difficult" when GPS is denied or only intermittently available, when little, if anything, is known about the environment, when communications are sporadic and unreliable or when operating conditions like lighting change unpredictably. And yet, somewhat perversely, it is often in just these conditions that our need to navigate is greatest: consider, for example, the surveying of buildings in a stricken nuclear facility such as Fukushima, or the autonomous driving of cars at night in cities where GPS coverage is poor. Intelligent navigation lies at the heart of much of mobile robotics research. It finds application in remote inspection, autonomous urban driving, defence, logistics, security and space robotics.

We shall consider how machines can acquire and manage the information they need to operate persistently in workspaces of our choosing. The goal is to demonstrate that performance improves through use and over time - something that comes naturally to humans and is immensely valuable in a machine. This goal poses questions about how the computers that control robots should represent their environment in a plastic fashion - one which can be stretched and pulled into different shapes over time. We also need to consider how to enable machines to decide how to act to improve their understanding of the world - alone and in concert with other vehicles, each with different sensors and capabilities. How can vehicle sensors be calibrated transparently and continuously? How can motion be planned to maximise both the coverage of inspections and the accuracy of workspace assessments? How can successful operation be guaranteed in the presence of unreliable or short-range communications?

This interweaving of the state of the art in navigation, planning and communications management is unusual and will allow us to ask and provide answers to challenging robotics science questions which, when exploited, will have a dramatic impact on the robots that will become indispensable in the future.

Planned Impact

The technical specifications of processors, sensors, networks and storage devices continue to improve at a staggering rate. Driven by this trend, in recent years autonomous systems research has made enormous strides towards applications
of significant value to the public domain. Robots are in the process of revolutionising the running of ports, mines, hospitals, factories and construction sites. Agents which can operate indefinitely will provide another step-change in the impact intelligent systems have on society. It will come in the shape of smarter transport and plant maintenance, lower cost logistics, more planetary science and increased security.

This proposal starts with state-of-the-art competencies in navigation, mapping, multi-vehicle coordination and data-flow management. It asks how these can be used in concert to address our thirst for labour-saving, efficiency-increasing, time-saving, security-giving robots and autonomous vehicles. It sets forth a research program built around the persistent acquisition, comprehension and exploitation of workspace models over indefinite periods, large scales and multiple agents.

To demonstrate impact we have several specific scenarios in mind which encompass a spectrum of autonomy:

1) A Mars rover sample-and-return mission which uses visual navigation to estimate its trajectory and, importantly, allows autonomous path retracing. In many ways planetary science epitomises impact in robotics - there is simply no alternative other than using robots.

2) A consumer, semi-autonomous road vehicle which, for navigation, exploits environmental knowledge acquired from a survey vehicle. In the face of varying lighting and weather conditions, this will be achieved using only a single, low-cost sensor. This approach is in stark contrast to the much vaunted google cars approach, which is always dependent on expensive 3D laser sensors.

3) A heterogeneous fleet of ground vehicles deployed in a reconnaissance role. They must not only navigate in changing conditions, but also plan to maximise the accuracy of their workspace characterisations (e.g. object detection performance) as well as maintaining communications connectivity.

4) Inspection of a nuclear plant or subsea well head with a tele-operated robot. We envision assistive autonomy in which a human operator remains in control of the vehicle but receives guidance on where the camera should be pointed next to ensure a complete and thorough inspection.

We stress that our view of impact is not limited to these four scenarios - they are simply illustrative. What we propose applies to any robotics scenario in which time and time again there is value in knowing "what is where" but one cannot depend on external infrastructure. We do not insist on prior knowledge - we can use it if available but, if not, we will accumulate what is needed autonomously. We do not insist on invariant workspaces - we set out to learn scene dynamics. We do not insist on good communications or superb object detectors - we plan to exploit the platforms' mobility to stay in touch while getting the best views of the world we can.

The mixing of planning, navigation, model-learning and communications management into a single proposal is an unusual, exciting and challenging prospect. It promises impact in the transport, logistics, defence, space and nuclear domains. There will never be fewer robots and what we propose here lies at the core of robotics science and its application to real world tasks.
 
Description Vehicles will only get smarter. There will always be a desire for more machine intelligence and autonomy. Our needs and expectations are ever increasing. As a result, we continue to pack more sensors and more computation into the robots that carry, transport, labour for and defend us. Here we interpret autonomy as a robot's ability to sense, understand and ultimately act of its own accord in its operating environment. This proposal was about giving autonomous vehicles the ability to navigate in difficult conditions over long periods of time. Conditions become "difficult" when GPS is denied or only intermittently available, when little, if anything, is known about the environment, when communications are sporadic and unreliable or when operating conditions like lighting change unpredictably. And yet, somewhat perversely, it is often in just these conditions that our need to navigate is greatest: consider, for example, the surveying of buildings in a stricken nuclear facility such as Fukushima, or the autonomous driving of cars at night in cities where GPS coverage is poor. Intelligent navigation lies at the heart of much of mobile robotics research. It finds application in remote inspection, autonomous urban driving, defence, logistics, security and space robotics. We considered how machines can acquire and manage the information they need to operate persistently in workspaces of our choosing. The goal was to demonstrate that performance improves through use and over time - something that comes naturally to humans and is immensely valuable in a machine. This goal posed questions about how the computers that control robots should represent their environment in a plastic fashion - one which can be stretched and pulled into different shapes over time - this resulted in a transformation paper called "plastic maps" by Churchill and Newman. We also needed to consider how to enable machines to operated robot. We envision assistive autonomy in which a human operator remains in control of the vehicle but receives guidance on where the camera should be pointed next to ensure a complete and thorough inspection. This has resulted in a new thread of work on dense reconstruction which will flow into the programme grant. We are due to start inspections of a fusion site within weeks. We stress that our view of impact has not been limited to these three scenarios - they are simply illustrative. What we propose applies to any robotics scenario in which time and time again there is value in knowing "what is where" but one cannot depend on external infrastructure. We do not insist on prior knowledge - we can use it if available but, if not, we will accumulate what is needed autonomously. We do not insist on invariant workspaces - we set out to learn scene dynamics and we have done so.
Exploitation Route The story of EPSRC's support for research that my involvement with continues with a £5 million Programme Grant that has just begun (March 2015). I am already looking at how big data could be harnessed in this area and will be linking up with some new industrial names for robotics exploitation.
And finally, this has all resulted in the creation of one of the UK's most exciting spin outs Oxbotica, which seeks to apply the science of mobile autonomy, to anything that moves on land. We will start with cars.
Sectors Digital/Communication/Information Technologies (including Software),Government, Democracy and Justice,Transport

URL http://ori.ox.ac.uk/
 
Description Driverless Cars
Geographic Reach National 
Policy Influence Type Gave evidence to a government review
URL http://www.policyexchange.org.uk/publications/category/item/eight-great-technologies
 
Description Life-Long Infrastructure Free Robot Navigation
Amount £1,655,490 (GBP)
Funding ID EP/I005021/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Academic/University
Country United Kingdom
Start 10/2010 
End 03/2016
 
Description Mobile Robotics: Enabling a Pervasive Technology of the Future
Amount £4,991,610 (GBP)
Funding ID EP/M019918/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Academic/University
Country United Kingdom
Start 03/2015 
End 02/2020
 
Description Nissan Collaboration 
Organisation Nissan Motor Company
Country Japan 
Sector Private 
PI Contribution 0.5M GBP income raise by a collaboration with Nissan Motor Company. This resulted in a demonstration of driverless car technology to the worlds media in Feb 2012. In October 2010 a collaborative deal was signed with Nissan Motor Company Inc. They are placing, at their own cost, an RA to work in my group and ?general support funding? for the group. Again, an I.P. deal is at the heart of the agreement . This has resulted in a demonstration of driverless technology in the UK See www.robotcar.org.uk
Start Year 2010
 
Title "NABU Man Portable Navigation and Survey Scanner" 
Description Asa Eckert-Erdheim, Scott Terry, Christopher Prahacs and Paul Newman "NABU Man Portable Navigation and Survey Scanner" 
IP Reference  
Protection Copyrighted (e.g. software)
Year Protection Granted
Licensed Yes
Impact Asa Eckert-Erdheim, Scott Terry, Christopher Prahacs and Paul Newman "NABU Man Portable Navigation and Survey Scanner"
 
Title "NABU4 Design" 
Description Chris Prahacs and Paul Newman "NABU4 Design" Not registered yet. Not licensed yet. 
IP Reference  
Protection Copyrighted (e.g. software)
Year Protection Granted
Licensed No
Impact Chris Prahacs and Paul Newman "NABU4 Design" Not registered yet. Not licensed yet.
 
Title Deep Image-based Detection 
Description Deep Image-based Detection 
IP Reference  
Protection Copyrighted (e.g. software)
Year Protection Granted 2016
Licensed Yes
Impact Deep Image-based Detection
 
Title Dense Laser Stereo 
Description Dense Laser Stereo 
IP Reference  
Protection Copyrighted (e.g. software)
Year Protection Granted 2016
Licensed Yes
Impact Dense Laser Stereo
 
Title Dub4 
Description Dub4 
IP Reference  
Protection Copyrighted (e.g. software)
Year Protection Granted 2016
Licensed Yes
Impact Dub4
 
Title Lidar Point Cloud 
Description Paul Newman and Ian Baldwin "Lidar Point Cloud" 
IP Reference  
Protection Patent application published
Year Protection Granted 2016
Licensed Yes
Impact Paul Newman and Ian Baldwin "Lidar Point Cloud"
 
Title OvO 
Description Visual Odometry System licensed for Mars Mission by ESA 
IP Reference  
Protection Copyrighted (e.g. software)
Year Protection Granted 2011
Licensed Yes
Impact Visual Odometry System licensed for Mars Mission by ESA
 
Title Parsing Traffic Lights Version 2 
Description Parsing Traffic Lights Version 2 
IP Reference  
Protection Copyrighted (e.g. software)
Year Protection Granted 2016
Licensed Yes
Impact Parsing Traffic Lights Version 2
 
Title Path Discovery using Random Forests and Dense Stereo 
Description Path Discovery using Random Forests and Dense Stereo 
IP Reference  
Protection Copyrighted (e.g. software)
Year Protection Granted 2016
Licensed Yes
Impact Path Discovery using Random Forests and Dense Stereo
 
Title Real-time Remote State Visualisation 
Description Real-time Remote State Visualisation 
IP Reference  
Protection Copyrighted (e.g. software)
Year Protection Granted 2016
Licensed Yes
Impact Real-time Remote State Visualisation
 
Title Scene Prior Builder v2 
Description Scene Prior Builder v2 
IP Reference  
Protection Copyrighted (e.g. software)
Year Protection Granted 2016
Licensed Yes
Impact Scene Prior Builder v2
 
Title Semantic Label Projection v2 
Description Semantic Label Projection v2 
IP Reference  
Protection Copyrighted (e.g. software)
Year Protection Granted 2016
Licensed Yes
Impact Semantic Label Projection v2
 
Title Semi-supervised Training for deep semantic Segmentation 
Description Semi-supervised Training for deep semantic Segmentation 
IP Reference  
Protection Copyrighted (e.g. software)
Year Protection Granted 2016
Licensed Yes
Impact Semi-supervised Training for deep semantic Segmentation
 
Title Vote3Deep 
Description Vote3Deep 
IP Reference  
Protection Copyrighted (e.g. software)
Year Protection Granted 2016
Licensed Yes
Impact Vote3Deep
 
Company Name Oxbotica 
Description Oxbotica is a spin-out from Oxford University's internationally acclaimed Mobile Robotics Group. We specialise in mobile autonomy, navigation and perception, and draw on our heritage of world leading research into autonomous robotics. Our solutions allow robots, vehicles, machinery and people to precisely map, navigate and actively interact with their surroundings, delivering new capability and precision to a wide range of applications. Our 3D imaging and localisation solutions operate indoors and outdoors and are suitable for use in applications ranging from hand held survey devices to autonomous vehicles. Oxbotica was founded by Prof. Ingmar Posner and Prof. Paul Newman - leaders of Oxford University's Mobile Robotics Group (MRG). MRG has an outstanding reputation for innovation and industrial collaborations (mrg.robots.ox.ac.uk). It has licensed navigation software for use on Mars rovers, developed the UK's first self-driving car, and has been a key and influential innovator in the area of Robotics and Autonomous Systems. 
Year Established 2014 
Impact Oxbotica will leverage the innovative and world leading outputs of the UK's premier mobile robotics group, enabling rapid commercialisation with our industry partners and further application of spin-off technologies.
Website http://www.oxbotica.com/
 
Company Name Navenio 
Description The company develops indoor positioning systems capable of localising people inside buildings, where GPS does not work. The Navenio technology relies on smartphone sensors and requires no additional infrastructure and no survey effort to be deployed in the building. It is therefore inherently scalable to a large number of buildings. The company focuses on applying the indoor positioning technology to a number of verticals, including retail and healthcare. 
Year Established 2015 
Impact * Live deployment of the technology in NHS trust; use of the technology for allocating tasks to porters based on their location * Current deployment and testing in several retail environments
Website https://www.navenio.com
 
Description BBC Radio 4 Interview - The Today Programme 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Public/other audiences
Results and Impact Go to 1:34:14 for Paul's interview on the Today Programme:
Year(s) Of Engagement Activity 2017
URL http://www.bbc.co.uk/programmes/b08hl5rt
 
Description EPSRC Robotics, Automation & Artificial Intelligence (RAAI) Theme Day 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Supporters
Results and Impact EPSRC are undertaking a review of our robotics, automation and artificial intelligence portfolios of relevance to Robotics and Autonomous Systems (RAS) in order to evaluate the quality and importance of EPSRC's portfolio of research and training in the area. To facilitate this we are hosting a Theme Day on the 31st January 2017 in Central London.

The Theme Day will involve poster presentations from holders of current and recent related grants from across the EPSRC portfolio. A panel of internationally leading experts chaired by Prof David Hogg will use the posters and discussions with attendees to draw conclusions about the portfolio as a whole. The outcomes of the review will be used to inform future strategy in the area of RAAI and will not impact on future funding decisions at a PI level.

The Theme day will be an opportunity for PIs to present their research to the review panel. The day will also give attendees an opportunity to view work of relevance to RAAI from across the EPSRC portfolio and to network with leaders in the area from across the UK.

As a holder of such a related grant(s) (details below) we would like to invite you to attend the event.

Related Grant(s): EP/I005021/1, EP/J012017/1, EP/M019918/1
Year(s) Of Engagement Activity 2017
 
Description ICRA 2015 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Submission of papers to ICRA 2015 - listed in publications
Year(s) Of Engagement Activity 2014,2015,2016
 
Description Michael Tanner (PhD student) judged the "regionals" of Vex Robotics competition (primarily a STEM-outreach program) hosted at Stowe School 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Schools
Results and Impact From: Michael Tanner
Subject: Vex Robotics Competition
Date: 6 February 2017 at 12:59:07 GMT
To: Paul Newman , Ingmar Posner

[This is purely an FYI/feel-good email. No action required.]

Paul/Ingmar,

Last week I took a day off to judge the "regionals" of Vex Robotics competition (primarily a STEM-outreach program) hosted at Stowe School
http://www.vexrobotics.com/vexedr/competition/

I thoroughly enjoyed the event and was fascinated at the creative designs students developed for this year's challenge. Here is a YouTube video showing the types of robots students develop:
https://www.youtube.com/watch?v=FCck9_vk8H4

The students were quite diverse (boys/girls, 10 - 17 years old, hail from UK/US/China, etc.), but they all shared a deep passion regarding their respective robot designs. The level of knowledge some of the students demonstrated was impressive (e.g., describing the chemical/material property trade-offs between various plastics included in their designs). Once the students learned I was studying robotics at Oxford, I was inuidated with questions.

I went home and immeditally ordered one of Vex Robotics' cheaper toy robotics kits (http://www.vexrobotics.com/vexiq) for my daughters to play with at home.

--
Michael Tanner
Oxford Robotics Institute
Department of Engineering Science
University of Oxford
Comm +44 7514 119187
Year(s) Of Engagement Activity 2017
URL http://www.vexrobotics.com/vexedr/competition/
 
Description RSS 2015 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Submission of papers to RSS 2015 - listed in publications
Year(s) Of Engagement Activity 2014,2015,2016
 
Description Shell Eco-Marathon June / July 2016 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact Shell Eco-marathon challenges student teams around the world to design, build, test and drive ultra-energy-efficient vehicles.
Year(s) Of Engagement Activity 2016,2017
URL https://www.youtube.com/watch?v=kU7OYLgnlkM
 
Description Talk at AHRC Research Network Workshop 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Professional Practitioners
Results and Impact The action-based alternative to 3D coordinate-frame representation was the main topic of our video meetings throughout 2016, and of our workshop at St John's College, Oxford, in January 2017. This workshop brought together experts from diverse disciplines for a focussed multidisciplinary discussion across three days, testing the action-based hypothesis by assessing its philosophical, computational and neuroscientific consequences. You can see the final discussion of the workshop here. As well as discussions led by Andrew and me, the workshop featured talks by:
Year(s) Of Engagement Activity 2017
URL https://jamesstazicker.com/research/the-action-based-brain/