Intelligent Workspace Acquisition, Comprehension and Exploitation for Mobile Autonomy in Infrastructure Denied Environments
Lead Research Organisation:
University of Oxford
Department Name: Engineering Science
Abstract
Vehicles will only get smarter. There will always be a desire for more machine intelligence and autonomy. Our needs and expectations are ever increasing. As a result, we continue to pack more sensors and more computation into the robots that carry, transport, labour for and defend us.
Here we interpret autonomy as a robot's ability to sense, understand and ultimately act of its own accord in its operating environment. This proposal is about giving autonomous vehicles the ability to navigate in difficult conditions over long periods of time. Conditions become "difficult" when GPS is denied or only intermittently available, when little, if anything, is known about the environment, when communications are sporadic and unreliable or when operating conditions like lighting change unpredictably. And yet, somewhat perversely, it is often in just these conditions that our need to navigate is greatest: consider, for example, the surveying of buildings in a stricken nuclear facility such as Fukushima, or the autonomous driving of cars at night in cities where GPS coverage is poor. Intelligent navigation lies at the heart of much of mobile robotics research. It finds application in remote inspection, autonomous urban driving, defence, logistics, security and space robotics.
We shall consider how machines can acquire and manage the information they need to operate persistently in workspaces of our choosing. The goal is to demonstrate that performance improves through use and over time - something that comes naturally to humans and is immensely valuable in a machine. This goal poses questions about how the computers that control robots should represent their environment in a plastic fashion - one which can be stretched and pulled into different shapes over time. We also need to consider how to enable machines to decide how to act to improve their understanding of the world - alone and in concert with other vehicles, each with different sensors and capabilities. How can vehicle sensors be calibrated transparently and continuously? How can motion be planned to maximise both the coverage of inspections and the accuracy of workspace assessments? How can successful operation be guaranteed in the presence of unreliable or short-range communications?
This interweaving of the state of the art in navigation, planning and communications management is unusual and will allow us to ask and provide answers to challenging robotics science questions which, when exploited, will have a dramatic impact on the robots that will become indispensable in the future.
Here we interpret autonomy as a robot's ability to sense, understand and ultimately act of its own accord in its operating environment. This proposal is about giving autonomous vehicles the ability to navigate in difficult conditions over long periods of time. Conditions become "difficult" when GPS is denied or only intermittently available, when little, if anything, is known about the environment, when communications are sporadic and unreliable or when operating conditions like lighting change unpredictably. And yet, somewhat perversely, it is often in just these conditions that our need to navigate is greatest: consider, for example, the surveying of buildings in a stricken nuclear facility such as Fukushima, or the autonomous driving of cars at night in cities where GPS coverage is poor. Intelligent navigation lies at the heart of much of mobile robotics research. It finds application in remote inspection, autonomous urban driving, defence, logistics, security and space robotics.
We shall consider how machines can acquire and manage the information they need to operate persistently in workspaces of our choosing. The goal is to demonstrate that performance improves through use and over time - something that comes naturally to humans and is immensely valuable in a machine. This goal poses questions about how the computers that control robots should represent their environment in a plastic fashion - one which can be stretched and pulled into different shapes over time. We also need to consider how to enable machines to decide how to act to improve their understanding of the world - alone and in concert with other vehicles, each with different sensors and capabilities. How can vehicle sensors be calibrated transparently and continuously? How can motion be planned to maximise both the coverage of inspections and the accuracy of workspace assessments? How can successful operation be guaranteed in the presence of unreliable or short-range communications?
This interweaving of the state of the art in navigation, planning and communications management is unusual and will allow us to ask and provide answers to challenging robotics science questions which, when exploited, will have a dramatic impact on the robots that will become indispensable in the future.
Planned Impact
The technical specifications of processors, sensors, networks and storage devices continue to improve at a staggering rate. Driven by this trend, in recent years autonomous systems research has made enormous strides towards applications
of significant value to the public domain. Robots are in the process of revolutionising the running of ports, mines, hospitals, factories and construction sites. Agents which can operate indefinitely will provide another step-change in the impact intelligent systems have on society. It will come in the shape of smarter transport and plant maintenance, lower cost logistics, more planetary science and increased security.
This proposal starts with state-of-the-art competencies in navigation, mapping, multi-vehicle coordination and data-flow management. It asks how these can be used in concert to address our thirst for labour-saving, efficiency-increasing, time-saving, security-giving robots and autonomous vehicles. It sets forth a research program built around the persistent acquisition, comprehension and exploitation of workspace models over indefinite periods, large scales and multiple agents.
To demonstrate impact we have several specific scenarios in mind which encompass a spectrum of autonomy:
1) A Mars rover sample-and-return mission which uses visual navigation to estimate its trajectory and, importantly, allows autonomous path retracing. In many ways planetary science epitomises impact in robotics - there is simply no alternative other than using robots.
2) A consumer, semi-autonomous road vehicle which, for navigation, exploits environmental knowledge acquired from a survey vehicle. In the face of varying lighting and weather conditions, this will be achieved using only a single, low-cost sensor. This approach is in stark contrast to the much vaunted google cars approach, which is always dependent on expensive 3D laser sensors.
3) A heterogeneous fleet of ground vehicles deployed in a reconnaissance role. They must not only navigate in changing conditions, but also plan to maximise the accuracy of their workspace characterisations (e.g. object detection performance) as well as maintaining communications connectivity.
4) Inspection of a nuclear plant or subsea well head with a tele-operated robot. We envision assistive autonomy in which a human operator remains in control of the vehicle but receives guidance on where the camera should be pointed next to ensure a complete and thorough inspection.
We stress that our view of impact is not limited to these four scenarios - they are simply illustrative. What we propose applies to any robotics scenario in which time and time again there is value in knowing "what is where" but one cannot depend on external infrastructure. We do not insist on prior knowledge - we can use it if available but, if not, we will accumulate what is needed autonomously. We do not insist on invariant workspaces - we set out to learn scene dynamics. We do not insist on good communications or superb object detectors - we plan to exploit the platforms' mobility to stay in touch while getting the best views of the world we can.
The mixing of planning, navigation, model-learning and communications management into a single proposal is an unusual, exciting and challenging prospect. It promises impact in the transport, logistics, defence, space and nuclear domains. There will never be fewer robots and what we propose here lies at the core of robotics science and its application to real world tasks.
of significant value to the public domain. Robots are in the process of revolutionising the running of ports, mines, hospitals, factories and construction sites. Agents which can operate indefinitely will provide another step-change in the impact intelligent systems have on society. It will come in the shape of smarter transport and plant maintenance, lower cost logistics, more planetary science and increased security.
This proposal starts with state-of-the-art competencies in navigation, mapping, multi-vehicle coordination and data-flow management. It asks how these can be used in concert to address our thirst for labour-saving, efficiency-increasing, time-saving, security-giving robots and autonomous vehicles. It sets forth a research program built around the persistent acquisition, comprehension and exploitation of workspace models over indefinite periods, large scales and multiple agents.
To demonstrate impact we have several specific scenarios in mind which encompass a spectrum of autonomy:
1) A Mars rover sample-and-return mission which uses visual navigation to estimate its trajectory and, importantly, allows autonomous path retracing. In many ways planetary science epitomises impact in robotics - there is simply no alternative other than using robots.
2) A consumer, semi-autonomous road vehicle which, for navigation, exploits environmental knowledge acquired from a survey vehicle. In the face of varying lighting and weather conditions, this will be achieved using only a single, low-cost sensor. This approach is in stark contrast to the much vaunted google cars approach, which is always dependent on expensive 3D laser sensors.
3) A heterogeneous fleet of ground vehicles deployed in a reconnaissance role. They must not only navigate in changing conditions, but also plan to maximise the accuracy of their workspace characterisations (e.g. object detection performance) as well as maintaining communications connectivity.
4) Inspection of a nuclear plant or subsea well head with a tele-operated robot. We envision assistive autonomy in which a human operator remains in control of the vehicle but receives guidance on where the camera should be pointed next to ensure a complete and thorough inspection.
We stress that our view of impact is not limited to these four scenarios - they are simply illustrative. What we propose applies to any robotics scenario in which time and time again there is value in knowing "what is where" but one cannot depend on external infrastructure. We do not insist on prior knowledge - we can use it if available but, if not, we will accumulate what is needed autonomously. We do not insist on invariant workspaces - we set out to learn scene dynamics. We do not insist on good communications or superb object detectors - we plan to exploit the platforms' mobility to stay in touch while getting the best views of the world we can.
The mixing of planning, navigation, model-learning and communications management into a single proposal is an unusual, exciting and challenging prospect. It promises impact in the transport, logistics, defence, space and nuclear domains. There will never be fewer robots and what we propose here lies at the core of robotics science and its application to real world tasks.
Publications
Abrudan T
(2016)
Underground Incrementally Deployed Magneto-Inductive 3-D Positioning Network
in IEEE Transactions on Geoscience and Remote Sensing
B. Mathibela
(2012)
Can Priors Be Trusted? Learning to Anticipate Roadworks
B. Mathibela
(2012)
Can Priors Be Trusted? Learning to Anticipate Roadworks
B. Upcroft
(2014)
Lighting Invariant Urban Street Classification
in Pending
Baldwin I
(2012)
Road vehicle localization with 2D push-broom LIDAR and 3D priors
C. McManus
(2014)
Shady Dealings: Robust, Long- Term Visual Localisation using Illumination Invariance
in Pending
C. McManus
(2014)
Scene Signatures: Localised and Point-less Features for Localisation
in Pending
Churchill W
(2013)
Experience-based navigation for long-term localisation
in The International Journal of Robotics Research
D. Z. Wang
(2013)
A New Approach to Model-Free Tracking with 2D Lidar
in Pending
D.Z. Wang
(2015)
Voting for Voting in Online Point Cloud Object Detection
Dominic Zeng Wang
(2012)
What could move? Finding cars, pedestrians and bicyclists in 3D laser data
G.Pascoe
(2015)
FARLAP: Fast Robust Localisation using Appearance Priors
Gadd Matthew
(2018)
The Data Market: Policies for Decentralised Visual Localisation
in arXiv e-prints
Description | Vehicles will only get smarter. There will always be a desire for more machine intelligence and autonomy. Our needs and expectations are ever increasing. As a result, we continue to pack more sensors and more computation into the robots that carry, transport, labour for and defend us. Here we interpret autonomy as a robot's ability to sense, understand and ultimately act of its own accord in its operating environment. This proposal was about giving autonomous vehicles the ability to navigate in difficult conditions over long periods of time. Conditions become "difficult" when GPS is denied or only intermittently available, when little, if anything, is known about the environment, when communications are sporadic and unreliable or when operating conditions like lighting change unpredictably. And yet, somewhat perversely, it is often in just these conditions that our need to navigate is greatest: consider, for example, the surveying of buildings in a stricken nuclear facility such as Fukushima, or the autonomous driving of cars at night in cities where GPS coverage is poor. Intelligent navigation lies at the heart of much of mobile robotics research. It finds application in remote inspection, autonomous urban driving, defence, logistics, security and space robotics. We considered how machines can acquire and manage the information they need to operate persistently in workspaces of our choosing. The goal was to demonstrate that performance improves through use and over time - something that comes naturally to humans and is immensely valuable in a machine. This goal posed questions about how the computers that control robots should represent their environment in a plastic fashion - one which can be stretched and pulled into different shapes over time - this resulted in a transformation paper called "plastic maps" by Churchill and Newman. We also needed to consider how to enable machines to operated robot. We envision assistive autonomy in which a human operator remains in control of the vehicle but receives guidance on where the camera should be pointed next to ensure a complete and thorough inspection. This has resulted in a new thread of work on dense reconstruction which will flow into the programme grant. We are due to start inspections of a fusion site within weeks. We stress that our view of impact has not been limited to these three scenarios - they are simply illustrative. What we propose applies to any robotics scenario in which time and time again there is value in knowing "what is where" but one cannot depend on external infrastructure. We do not insist on prior knowledge - we can use it if available but, if not, we will accumulate what is needed autonomously. We do not insist on invariant workspaces - we set out to learn scene dynamics and we have done so. |
Exploitation Route | The story of EPSRC's support for research that my involvement with continues with a £5 million Programme Grant that has just begun (March 2015). I am already looking at how big data could be harnessed in this area and will be linking up with some new industrial names for robotics exploitation. And finally, this has all resulted in the creation of one of the UK's most exciting spin outs Oxbotica, which seeks to apply the science of mobile autonomy, to anything that moves on land. We will start with cars. |
Sectors | Digital/Communication/Information Technologies (including Software) Government Democracy and Justice Transport |
URL | http://ori.ox.ac.uk/ |
Description | Life-Long Infrastructure Free Robot Navigation |
Amount | £1,655,490 (GBP) |
Funding ID | EP/I005021/1 |
Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
Sector | Public |
Country | United Kingdom |
Start | 09/2010 |
End | 03/2016 |
Description | Mobile Robotics: Enabling a Pervasive Technology of the Future |
Amount | £4,991,610 (GBP) |
Funding ID | EP/M019918/1 |
Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
Sector | Public |
Country | United Kingdom |
Start | 03/2015 |
End | 02/2020 |
Description | Nissan Collaboration |
Organisation | Nissan Motor Company |
Country | Japan |
Sector | Private |
PI Contribution | 0.5M GBP income raise by a collaboration with Nissan Motor Company. This resulted in a demonstration of driverless car technology to the worlds media in Feb 2012. In October 2010 a collaborative deal was signed with Nissan Motor Company Inc. They are placing, at their own cost, an RA to work in my group and ?general support funding? for the group. Again, an I.P. deal is at the heart of the agreement . This has resulted in a demonstration of driverless technology in the UK See www.robotcar.org.uk |
Start Year | 2010 |
Title | "NABU Man Portable Navigation and Survey Scanner" |
Description | Asa Eckert-Erdheim, Scott Terry, Christopher Prahacs and Paul Newman "NABU Man Portable Navigation and Survey Scanner" |
IP Reference | |
Protection | Copyrighted (e.g. software) |
Year Protection Granted | |
Licensed | Yes |
Impact | Asa Eckert-Erdheim, Scott Terry, Christopher Prahacs and Paul Newman "NABU Man Portable Navigation and Survey Scanner" |
Title | "NABU4 Design" |
Description | Chris Prahacs and Paul Newman "NABU4 Design" Not registered yet. Not licensed yet. |
IP Reference | |
Protection | Copyrighted (e.g. software) |
Year Protection Granted | |
Licensed | No |
Impact | Chris Prahacs and Paul Newman "NABU4 Design" Not registered yet. Not licensed yet. |
Title | Deep Image-based Detection |
Description | Deep Image-based Detection |
IP Reference | |
Protection | Copyrighted (e.g. software) |
Year Protection Granted | 2016 |
Licensed | Yes |
Impact | Deep Image-based Detection |
Title | Dense Laser Stereo |
Description | Dense Laser Stereo |
IP Reference | |
Protection | Copyrighted (e.g. software) |
Year Protection Granted | 2016 |
Licensed | Yes |
Impact | Dense Laser Stereo |
Title | Dub4 |
Description | Dub4 |
IP Reference | |
Protection | Copyrighted (e.g. software) |
Year Protection Granted | 2016 |
Licensed | Yes |
Impact | Dub4 |
Title | Lidar Point Cloud |
Description | Paul Newman and Ian Baldwin "Lidar Point Cloud" |
IP Reference | |
Protection | Patent application published |
Year Protection Granted | 2016 |
Licensed | Yes |
Impact | Paul Newman and Ian Baldwin "Lidar Point Cloud" |
Title | OvO |
Description | Visual Odometry System licensed for Mars Mission by ESA |
IP Reference | |
Protection | Copyrighted (e.g. software) |
Year Protection Granted | 2011 |
Licensed | Yes |
Impact | Visual Odometry System licensed for Mars Mission by ESA |
Title | Parsing Traffic Lights Version 2 |
Description | Parsing Traffic Lights Version 2 |
IP Reference | |
Protection | Copyrighted (e.g. software) |
Year Protection Granted | 2016 |
Licensed | Yes |
Impact | Parsing Traffic Lights Version 2 |
Title | Path Discovery using Random Forests and Dense Stereo |
Description | Path Discovery using Random Forests and Dense Stereo |
IP Reference | |
Protection | Copyrighted (e.g. software) |
Year Protection Granted | 2016 |
Licensed | Yes |
Impact | Path Discovery using Random Forests and Dense Stereo |
Title | Real-time Remote State Visualisation |
Description | Real-time Remote State Visualisation |
IP Reference | |
Protection | Copyrighted (e.g. software) |
Year Protection Granted | 2016 |
Licensed | Yes |
Impact | Real-time Remote State Visualisation |
Title | Scene Prior Builder v2 |
Description | Scene Prior Builder v2 |
IP Reference | |
Protection | Copyrighted (e.g. software) |
Year Protection Granted | 2016 |
Licensed | Yes |
Impact | Scene Prior Builder v2 |
Title | Semantic Label Projection v2 |
Description | Semantic Label Projection v2 |
IP Reference | |
Protection | Copyrighted (e.g. software) |
Year Protection Granted | 2016 |
Licensed | Yes |
Impact | Semantic Label Projection v2 |
Title | Semi-supervised Training for deep semantic Segmentation |
Description | Semi-supervised Training for deep semantic Segmentation |
IP Reference | |
Protection | Copyrighted (e.g. software) |
Year Protection Granted | 2016 |
Licensed | Yes |
Impact | Semi-supervised Training for deep semantic Segmentation |
Title | Vote3Deep |
Description | Vote3Deep |
IP Reference | |
Protection | Copyrighted (e.g. software) |
Year Protection Granted | 2016 |
Licensed | Yes |
Impact | Vote3Deep |
Company Name | Navenio |
Description | Navenio develops software which tracks the location and path of people within an indoor environment, via an app and mobile sensors. This data can be used in retail, healthcare and transport-hub environments, to improve efficiency in the movement of people and marketing of products. |
Year Established | 2015 |
Impact | * Live deployment of the technology in NHS trust; use of the technology for allocating tasks to porters based on their location * Current deployment and testing in several retail environments |
Website | http://www.navenio.com |
Company Name | Oxa |
Description | Oxa develops software designed to power driverless vehicles, with technology that uses features such as cameras and lasers in order to sense and navigate the surrounding environment. |
Year Established | 2014 |
Impact | Oxbotica will leverage the innovative and world leading outputs of the UK's premier mobile robotics group, enabling rapid commercialisation with our industry partners and further application of spin-off technologies. |
Website | http://www.oxbotica.com |
Description | BBC Radio 4 Interview - The Today Programme |
Form Of Engagement Activity | A press release, press conference or response to a media enquiry/interview |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Public/other audiences |
Results and Impact | Go to 1:34:14 for Paul's interview on the Today Programme: |
Year(s) Of Engagement Activity | 2017 |
URL | http://www.bbc.co.uk/programmes/b08hl5rt |
Description | EPSRC Robotics, Automation & Artificial Intelligence (RAAI) Theme Day |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Supporters |
Results and Impact | EPSRC are undertaking a review of our robotics, automation and artificial intelligence portfolios of relevance to Robotics and Autonomous Systems (RAS) in order to evaluate the quality and importance of EPSRC's portfolio of research and training in the area. To facilitate this we are hosting a Theme Day on the 31st January 2017 in Central London. The Theme Day will involve poster presentations from holders of current and recent related grants from across the EPSRC portfolio. A panel of internationally leading experts chaired by Prof David Hogg will use the posters and discussions with attendees to draw conclusions about the portfolio as a whole. The outcomes of the review will be used to inform future strategy in the area of RAAI and will not impact on future funding decisions at a PI level. The Theme day will be an opportunity for PIs to present their research to the review panel. The day will also give attendees an opportunity to view work of relevance to RAAI from across the EPSRC portfolio and to network with leaders in the area from across the UK. As a holder of such a related grant(s) (details below) we would like to invite you to attend the event. Related Grant(s): EP/I005021/1, EP/J012017/1, EP/M019918/1 |
Year(s) Of Engagement Activity | 2017 |
Description | ICRA 2015 |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Submission of papers to ICRA 2015 - listed in publications |
Year(s) Of Engagement Activity | 2014,2015,2016 |
Description | Michael Tanner (PhD student) judged the "regionals" of Vex Robotics competition (primarily a STEM-outreach program) hosted at Stowe School |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | Regional |
Primary Audience | Schools |
Results and Impact | From: Michael Tanner Subject: Vex Robotics Competition Date: 6 February 2017 at 12:59:07 GMT To: Paul Newman , Ingmar Posner [This is purely an FYI/feel-good email. No action required.] Paul/Ingmar, Last week I took a day off to judge the "regionals" of Vex Robotics competition (primarily a STEM-outreach program) hosted at Stowe School http://www.vexrobotics.com/vexedr/competition/ I thoroughly enjoyed the event and was fascinated at the creative designs students developed for this year's challenge. Here is a YouTube video showing the types of robots students develop: https://www.youtube.com/watch?v=FCck9_vk8H4 The students were quite diverse (boys/girls, 10 - 17 years old, hail from UK/US/China, etc.), but they all shared a deep passion regarding their respective robot designs. The level of knowledge some of the students demonstrated was impressive (e.g., describing the chemical/material property trade-offs between various plastics included in their designs). Once the students learned I was studying robotics at Oxford, I was inuidated with questions. I went home and immeditally ordered one of Vex Robotics' cheaper toy robotics kits (http://www.vexrobotics.com/vexiq) for my daughters to play with at home. -- Michael Tanner Oxford Robotics Institute Department of Engineering Science University of Oxford Comm +44 7514 119187 |
Year(s) Of Engagement Activity | 2017 |
URL | http://www.vexrobotics.com/vexedr/competition/ |
Description | RSS 2015 |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Submission of papers to RSS 2015 - listed in publications |
Year(s) Of Engagement Activity | 2014,2015,2016 |
Description | Shell Eco-Marathon June / July 2016 |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | Shell Eco-marathon challenges student teams around the world to design, build, test and drive ultra-energy-efficient vehicles. |
Year(s) Of Engagement Activity | 2016,2017 |
URL | https://www.youtube.com/watch?v=kU7OYLgnlkM |
Description | Talk at AHRC Research Network Workshop |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Regional |
Primary Audience | Professional Practitioners |
Results and Impact | The action-based alternative to 3D coordinate-frame representation was the main topic of our video meetings throughout 2016, and of our workshop at St John's College, Oxford, in January 2017. This workshop brought together experts from diverse disciplines for a focussed multidisciplinary discussion across three days, testing the action-based hypothesis by assessing its philosophical, computational and neuroscientific consequences. You can see the final discussion of the workshop here. As well as discussions led by Andrew and me, the workshop featured talks by: |
Year(s) Of Engagement Activity | 2017 |
URL | https://jamesstazicker.com/research/the-action-based-brain/ |