Exploiting invisible cues for robot navigation in complex natural environments
Lead Research Organisation:
University of Edinburgh
Department Name: Sch of Informatics
Abstract
Outdoor navigation in natural environments remains a challenge for robotics. Recent breakthroughs in robot navigation have been dependent on specific sensor technologies, such as laser depth sensors and GPS, and advanced image processing. The ability of animals such as ants to navigate effectively without such power- and computation- hungry systems are a proof of principle that alternative cheaper approaches are viable. Ants also have specialised sensing, with a peripheral visual system that has evolved to be sensitive to crucial cues for navigation. Specifically, they make use of non-visible (to humans) light cues in the form of ultraviolet (UV) and polarised light detection. UV detection allows the important signal of the horizon shape against the sky to be easily distinguished. Polarised light detection provides an external compass cue of heading relative to the sun direction, even when only a small portion of the sky is visible.
We propose to build a sensory system that gathers the full range of light cues available to the ant, in its natural ecological situation, and to analyse the information contained in this signal. We will also analyse how the specific sensor layout (ommatidia array), peripheral receptor characteristics, and the motor behaviour of the ant may contribute to extracting salient information. The data will form a test-bed for comparison of algorithmic and neural models of the processing that underlies the navigation capabilities of the ant. To date, these cues have been considered separately but we believe the navigational success of this system depends on the specific combination. For example, the directional information in the polarised sky may form an important part of visual memories; and UV information may contribute to disambiguation of the polarisation pattern and the robustness of this information under different cloud conditions.
There has been a substantial increase in the last few years in research into insect neural pathways involved in processing these cues which has yet to be exploited in robot models. In particular there has been breakthrough work on the central brain mechanisms involved in decoding polarised light to obtain heading direction. There is also a rapidly increasing understanding of the circuits involved in learning, a key component of navigation capabilities.
A key aspect of our method, in comparison to many bio-inspired projects, is that we do not propose to start by designing and manufacturing a sensor as similar as possible to the ant eye. Rather, we consider it essential to first understand what aspects of the eye are actually key to support navigation, and what variation from the exact biological design will conserve the desirable properties while simplifying the manufacture. The final outcome of this research will be a detailed design for a small scale and low power vision system for outdoor navigation. We hypothesis that we can leverage the miniaturisation of CCD cameras (which are normally sensitive to the upper range of UV, but have filters applied to remove it) to compose an omnidirectional compound eye where the filter characteristics of each ommatidia are appropriately tuned for wavelength or polarisation plane, in imitation of the ant eye. Beyond the direct application to robot navigation in natural environments, in tasks such as agriculture, search and rescue, and environmental clean-up, such a design has potential to be developed for other applications, including mobile devices, inspection and surveillance.
We propose to build a sensory system that gathers the full range of light cues available to the ant, in its natural ecological situation, and to analyse the information contained in this signal. We will also analyse how the specific sensor layout (ommatidia array), peripheral receptor characteristics, and the motor behaviour of the ant may contribute to extracting salient information. The data will form a test-bed for comparison of algorithmic and neural models of the processing that underlies the navigation capabilities of the ant. To date, these cues have been considered separately but we believe the navigational success of this system depends on the specific combination. For example, the directional information in the polarised sky may form an important part of visual memories; and UV information may contribute to disambiguation of the polarisation pattern and the robustness of this information under different cloud conditions.
There has been a substantial increase in the last few years in research into insect neural pathways involved in processing these cues which has yet to be exploited in robot models. In particular there has been breakthrough work on the central brain mechanisms involved in decoding polarised light to obtain heading direction. There is also a rapidly increasing understanding of the circuits involved in learning, a key component of navigation capabilities.
A key aspect of our method, in comparison to many bio-inspired projects, is that we do not propose to start by designing and manufacturing a sensor as similar as possible to the ant eye. Rather, we consider it essential to first understand what aspects of the eye are actually key to support navigation, and what variation from the exact biological design will conserve the desirable properties while simplifying the manufacture. The final outcome of this research will be a detailed design for a small scale and low power vision system for outdoor navigation. We hypothesis that we can leverage the miniaturisation of CCD cameras (which are normally sensitive to the upper range of UV, but have filters applied to remove it) to compose an omnidirectional compound eye where the filter characteristics of each ommatidia are appropriately tuned for wavelength or polarisation plane, in imitation of the ant eye. Beyond the direct application to robot navigation in natural environments, in tasks such as agriculture, search and rescue, and environmental clean-up, such a design has potential to be developed for other applications, including mobile devices, inspection and surveillance.
Planned Impact
UK and European governments have highlighted the robot industry as a key research and growth sector. Currently the market is still dominated by industrial robots confined to factories. Robotic cars have recently shown that mobile autonomy outdoors is possible yet are reliant on energy and computationally expensive sensing (e.g. 3D laser scanners). Development of a novel low-cost sensor for robot navigation is immediately applicable to this emerging market. Mid-term applications for the system include self-guided systems; robots in agriculture; exploration; environmental monitoring and clean-up operations. We also predict there will be potential applications other than robotics for miniaturised, omnidirectional UV and polarisation sensing.
The outcome of this proposal will be a prototype system with a design for manufacture, and patenting possibilities. We will fully investigate the options for commercialisation either through a university spin out or a licencing agreement with industrial partners. Primary market research shall be conducted by attending targeted industrial events at which companies likely to be interested in autonomous navigation (e.g. specialist robot developers, automotive manufacturers, defence contractors) are in attendance. Looking beyond the specific device we propose to develop here, we also hope to demonstrate to industry the real benefit of exploiting natural solutions for specialised sensing problems, to increase direct investment in future research in biomimetics through industrial-academic partnerships. The work undertaken here will provide training and experience in commercialisation to the junior project participants and may ultimately contribute to continued expansion and employment opportunities in this area.
The key industries where we expect impact are areas of social importance and the approach can contribute in the long term to environmental sustainability, safety and energy reduction goals. In contrast to the conventional public image of robotics as aiming for human-like machines, the research is premised on building small cheap devices that may make a big difference. We look to involve the public and stakeholders in this emerging technology through a series of science communication activities, as detailed in the Pathways to Impact.
The outcome of this proposal will be a prototype system with a design for manufacture, and patenting possibilities. We will fully investigate the options for commercialisation either through a university spin out or a licencing agreement with industrial partners. Primary market research shall be conducted by attending targeted industrial events at which companies likely to be interested in autonomous navigation (e.g. specialist robot developers, automotive manufacturers, defence contractors) are in attendance. Looking beyond the specific device we propose to develop here, we also hope to demonstrate to industry the real benefit of exploiting natural solutions for specialised sensing problems, to increase direct investment in future research in biomimetics through industrial-academic partnerships. The work undertaken here will provide training and experience in commercialisation to the junior project participants and may ultimately contribute to continued expansion and employment opportunities in this area.
The key industries where we expect impact are areas of social importance and the approach can contribute in the long term to environmental sustainability, safety and energy reduction goals. In contrast to the conventional public image of robotics as aiming for human-like machines, the research is premised on building small cheap devices that may make a big difference. We look to involve the public and stakeholders in this emerging technology through a series of science communication activities, as detailed in the Pathways to Impact.
Publications
Ardin PB
(2016)
Ant Homing Ability Is Not Diminished When Traveling Backwards.
in Frontiers in behavioral neuroscience
Webb B
(2016)
Neural mechanisms of insect navigation.
in Current opinion in insect science
Ardin P
(2016)
Using an Insect Mushroom Body Circuit to Encode Route Memory in Complex Natural Environments.
in PLoS computational biology
Schwarz S
(2017)
How Ants Use Vision When Homing Backward.
in Current biology : CB
Stone T
(2017)
An Anatomically Constrained Model for Path Integration in the Bee Brain.
in Current biology : CB
Description | We have developed a model of processing of polarised light in the insect eye and brain, and shown how this can effectively extract compass information from the pattern in skylight, which forms the basis of a proposed new design for a robot sky compass sensor. The output has been incorporated into neural algorithms, inspired by the ant brain, that can explain insect navigation behaviour and can be used on robots. The models have been tested in a realistic reconstruction of the ant's natural environment, and compared to results from tracking ants in natural terrain. This was supported by the development of general software tools allowing meshing of natural scenes from laser scan data, and tracking of animals using a handheld camera under natural conditions. We have also shown how the use of ultraviolet light in conjunction with a spherical harmonic encoding method can improve robot localisation based on sky segmentation under challenging conditions. We have carried out new behavioural experiments showing that ants are capable of navigating backwards and can transfer information between their terrestrial visual memory and celestial compass. |
Exploitation Route | The methods developed should be applicable to navigation technology, particularly for low-cost systems in natural contexts, such as environmental monitoring and agriculture. The neural models we have developed are also of general interest to understand cognition in insects. |
Sectors | Agriculture Food and Drink Digital/Communication/Information Technologies (including Software) Electronics Environment Transport |
Description | Some findings from this research are now being translated to use on robot platforms in industry settings. Specifically, the start-up company Opteran Technologies are taking forward ideas for visual processing methods explored in the project to use on indoor and outdoor robot systems. |
First Year Of Impact | 2020 |
Sector | Aerospace, Defence and Marine,Environment,Retail,Transport |
Impact Types | Economic |
Description | 3B: brains beat brawn |
Amount | £1,287,730 (GBP) |
Funding ID | 900305 |
Organisation | United Kingdom Research and Innovation |
Sector | Public |
Country | United Kingdom |
Start | 03/2024 |
End | 02/2028 |
Description | AHDB PhD Studentship |
Amount | £70,388 (GBP) |
Organisation | Agricultural and Horticulture Development Board |
Sector | Charity/Non Profit |
Country | United Kingdom |
Start | 07/2017 |
End | 09/2020 |
Description | Discovery Projects |
Amount | $483,000 (AUD) |
Funding ID | DP160102658 |
Organisation | Australian Research Council |
Sector | Public |
Country | Australia |
Start | 01/2016 |
End | 12/2018 |
Description | EPSRC Impact Acceleration Account |
Amount | £36,898 (GBP) |
Funding ID | PIV068 |
Organisation | University of Edinburgh |
Sector | Academic/University |
Country | United Kingdom |
Start | 06/2021 |
End | 03/2022 |
Description | Guarantee funding via Innovate UK for European Commission EIC Pathfinder award |
Amount | £200,000 (GBP) |
Funding ID | 10032249 |
Organisation | Innovate UK |
Sector | Public |
Country | United Kingdom |
Start | 03/2022 |
End | 03/2026 |
Description | Responsive mode |
Amount | £286,057 (GBP) |
Funding ID | BB/R005052/1 |
Organisation | Biotechnology and Biological Sciences Research Council (BBSRC) |
Sector | Public |
Country | United Kingdom |
Start | 01/2018 |
End | 12/2020 |
Description | SkyEye: Feasibility Study of Vision Based Localisation in GPS-compromised Environments |
Amount | £24,993 (GBP) |
Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
Sector | Public |
Country | United Kingdom |
Start | 01/2022 |
End | 03/2022 |
Description | UltimateCOMPASS ERC consolidator grant |
Amount | € 188,924 (EUR) |
Organisation | European Commission |
Sector | Public |
Country | European Union (EU) |
Start | 06/2019 |
End | 06/2024 |
Title | Habitat3D |
Description | Habtat3D is an open source pipeline that converts multiple point counts into a mesh allowing visualisation, adaptioation, and analysis |
Type Of Material | Improvements to research infrastructure |
Year Produced | 2018 |
Provided To Others? | Yes |
Impact | We have had contact from researchers within and outside of our field looking to use this tool for their own research. |
URL | http://www.insectvision.org/3d-reconstruction-tools/habitat3d |
Title | Habitracks |
Description | Habitracks is a software tool for researchers looking to track animals in their natural habitats. |
Type Of Material | Improvements to research infrastructure |
Year Produced | 2017 |
Provided To Others? | Yes |
Impact | This tool has already been adopted by a number of international laboratories for use in their behavioural experiments. |
URL | http://openaccess.thecvf.com/content_ICCV_2017_workshops/papers/w41/Risse_Visual_Tracking_of_ICCV_20... |
Title | roboant |
Description | Robot platform for algorithm prototyping |
Type Of Material | Technology assay or reagent |
Year Produced | 2017 |
Provided To Others? | Yes |
Impact | This platform has been used by a number of students for various projects. |
URL | https://blog.inf.ed.ac.uk/insectrobotics/roboant/ |
Title | 3D Mesh of ant habitat in Canberra, Australia |
Description | Reconstruction of an ant habitat (Myrmecia croslandi) based on 9 clouds captured in Canberra, Australia. The 8800sqm area features several complex Eucalyptus trees. This reconstruction was tuned to demonstrate compressibility features of Habitat3D: While strongly compressing the ground (>99% size reduction; file: ground.ply) all characteristics of the complex trees are preserved (file: vegetation.ply). |
Type Of Material | Database/Collection of data |
Year Produced | 2017 |
Provided To Others? | Yes |
Impact | This environment forms a cornerstone of attempts to assess how insects visually navigate their environments. Combined with additional tools research groups will be able to benchmark their models using this data. |
URL | http://www.insectvision.org/3d-reconstruction-tools/habitat3d |
Title | 3D mesh of ant habitat in Seville, Spain |
Description | Reconstruction of a natural foraging environment of desert ants (Cataglyphis velox) based on 56 laser scans and covering an 1018sqm area. This highly cluttered terrain features hundreds of individual plants and the reconstruction was tuned towards overall accuracy: Both, the ground and vegetation reconstruction feature millimetre accuracy while reducing the memory load to 1.5%. The reconstruction is divided into vegetation (inner_vegetation.ply and outer_vegetation.ply) and ground (inner_ground.ply and outer_ground.ply). |
Type Of Material | Database/Collection of data |
Year Produced | 2017 |
Provided To Others? | Yes |
Impact | This environment forms a cornerstone of attempts to assess how insects visually navigate their environments. Combined with additional tools research groups will be able to benchmark their models using this data. |
URL | http://www.insectvision.org/3d-reconstruction-tools/habitat3d |
Title | Desert Ant Ontogeny Dataset |
Description | The entire foraging life of desert ant documented in a series of videos, provided with the tracking software, the tracks, and an environment reconstruction |
Type Of Material | Database/Collection of data |
Year Produced | 2023 |
Provided To Others? | Yes |
Impact | New insights were made in the publication associated with the dataset that are driving new research questions. |
URL | https://cater.cvmls.org/ |
Title | Wildlife Animal Tracking (WAT) dataset |
Description | A dataset of small animals tracked in their natural habitats |
Type Of Material | Database/Collection of data |
Year Produced | 2017 |
Provided To Others? | Yes |
Impact | This dataset presents a new benchmark for new tracking software |
URL | http://blog.inf.ed.ac.uk/insectrobotics/WAT |
Description | Lund |
Organisation | Lund University |
Country | Sweden |
Sector | Academic/University |
PI Contribution | Computational modelling of path integration in the central complex of the bee. |
Collaborator Contribution | Neurophysiological and neuroanatomical data on the central complex |
Impact | Paper in Current Biology (multidisciplinary: biology, computing, robotics) Two proposals (in submission/preparation) for funding from the Swedish government to explore the transfer of ideas from insect neural circuits to optical computing with nano components (multidisciplinary: biology, computing, physics, engineering) |
Start Year | 2015 |
Description | Lund2 |
Organisation | Lund University |
Country | Sweden |
Sector | Academic/University |
PI Contribution | I have supervised several projects that implement the behaviour of the Dung beetle on a robot using a neural model we devised. |
Collaborator Contribution | The partner at Lund, Marie Dacke, has contributed data and advice to the student projects. |
Impact | A grant from ERC to support further work has been awarded. This is a multidisciplinary collaboration, combining behavioural biology, neuroscience, modelling and robotics. |
Start Year | 2016 |
Description | Modelling the honeybee polarization compass |
Organisation | University of Wurzburg |
Country | Germany |
Sector | Academic/University |
PI Contribution | Providing the computational model and collaborating on its adaptation to data from the honeybee |
Collaborator Contribution | Providing data on the honeybee |
Impact | The partner at Wurzburg was awarded a DFG grant to support this work. The collaboration was specifically described as part of the successful proposal. |
Start Year | 2020 |
Description | Sussex |
Organisation | University of Sussex |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | Joint work on behavioural experiments on ants which is now linked to model development and a funded grant proposal |
Collaborator Contribution | Behavioural and neural methods for ant experiments |
Impact | Multidisciplinary - robotics and biology |
Start Year | 2016 |
Description | Toulouse |
Organisation | University of Toulouse |
Country | France |
Sector | Academic/University |
PI Contribution | Joint work on behavioural experiments in desert ants |
Collaborator Contribution | Joint work on behavioural experiments in desert ants |
Impact | There have been a number of journal articles, posters, and talks directly resulting from this collaboration. All are listed in the appropriate sections. |
Start Year | 2016 |
Title | CATER: combined animal tracking and environment reconstruction |
Description | Method to extract animal positions from video data and to embed those tracks in a reconstructed background allowing high spatiotemporal analysis of animal behaviour in the wild and with reference to the habitat. |
Type Of Technology | New/Improved Technique/Technology |
Year Produced | 2024 |
Open Source License? | Yes |
Impact | Insights raised from analysis linked to methods paper Multiple lab groups now using the tool New studies have been inspired by these outcomes. |
URL | https://www.science.org/doi/10.1126/sciadv.adg2094 |
Title | Habitat3D |
Description | Habitat3D is an open source cross-platform tool to generate photorealistic meshes from point clouds of natural outdoor scenes. All necessary processing steps (filtering, segmentation, feature extraction, meshing, etc.) as well as recipe-based generation of pipelines are incorporated in a GUI-based framework. Habitat3D is written in C++ and requires Qt, PCL, VTK and BOOST. |
Type Of Technology | Software |
Year Produced | 2016 |
Open Source License? | Yes |
Impact | The tool has been used to recreate ant environments from our own and others field studies |
URL | http://www.insectvision.org/3d-reconstruction-tools/habitat3d |
Title | Habitracks |
Description | Habitracks provides a tool for tracking small objects in images taken with a moving camera |
Type Of Technology | Software |
Year Produced | 2017 |
Open Source License? | Yes |
Impact | This software is already being adopted by other researchers in the field |
URL | http://openaccess.thecvf.com/content_ICCV_2017_workshops/papers/w41/Risse_Visual_Tracking_of_ICCV_20... |
Title | Skycompass |
Description | We have constructed a physical model of the sky-compass investigated in this project. This consists of 8 pairs of UV-sensitive photodiodes behind orthogonal polarising filters, with custom designed circuit boards to read out the signals. The units are arranged in a specific pattern to point at different areas of the sky, and mounted on a robot which can rotate to collect data at different orientations and under different outdoor conditions. |
Type Of Technology | Physical Model/Kit |
Year Produced | 2022 |
Impact | A paper is in preparation to describe the work, and we are investigating miniaturisation of the device to make it suitable for unmanned aerial vehicles. |
Description | Backwards ants press coverage |
Form Of Engagement Activity | A press release, press conference or response to a media enquiry/interview |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Media (as a channel to the public) |
Results and Impact | A press release was made describing the results of our Current Biology paper "How Ants Use Vision When Homing Backward". The story was picked up internationally and appeared in the following venues: DAILY MAIL, DAILY MIRROR, THE HERALD, WESTERN DAILY PRESS, WIRED.CO.UK, AFP, EUROPE 1, L'EXPRESS.FR, FRANCE INFO, RTL TVI, LIBERATION, LE POINT (France) ATS, LE JOURNAL DU JURA (Switzerland) APS (Algeria) LA NOUVELLE GAZETTE, LA MEUSE, NORD ÉCLAIR, VRTNIEUWS.NET (Belgium) MONTEVIDEO PORTAL (Uruguay) DIARIO UNO (Argentina) EL FINANCIERO (Mexico) DIARIO MONTANES, LA VANGUARDIA, LA VOZ DIGITAL, ABC.ES, EUROPA PRESS, EL ESPECTADOR (Spain) IRISH DAILY MAIL (Ireland) PEOPLE'S DAILY ONLINE, PUEBLO EN LINEA, CHINA POST, CHINA TIMES, CAN, AFP, WORKER CHINA, KAN KAN NEWS (China) BERLINGSKE (Denmark) CHRISTIAN SCIENCE MONITOR, TERRA DAILY, DISCOVERY CHANNEL SCIENCE NEWS, TECH TIMES (USA) EXPATICA (Netherlands) ENCA, ARGUS, NEWS24, (S Africa) NEW VISION (Uganda) THE JAPAN TIMES (Japan) IRAN DAILY (Iran) PAGINA SIETE (Bolivia) WEBINDIA123.COM, INDO ASIAN NEWS SERVICE, SIASAT DAILY, NETINDIA123.COM, DNA INDIA, BUSINESS STANDARD (India) STRAITS TIMES (Singapore) YAM NEWS, EPOCH TIMES (Taiwan) CANADIAN BROADCASTING CORP (Canada) ZME SCIENCE (Romania). |
Year(s) Of Engagement Activity | 2017 |
Description | Embodied AI podcast |
Form Of Engagement Activity | Engagement focused website, blog or social media channel |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Public/other audiences |
Results and Impact | Barbara Webb was interviewed for the The Embodied AI Podcast, promoted by the British Neuroscience Association. |
Year(s) Of Engagement Activity | 2022 |
URL | https://anchor.fm/the-embodied-ai-podcast |
Description | Presentation at New Scientist Live |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Public/other audiences |
Results and Impact | New Scientist Live is an award-winning festival of ideas and discoveries for everyone curious about science and why it matters, held over five days in London. |
Year(s) Of Engagement Activity | 2017 |
Description | Presentation at VIIHM "Vision for movement" workshop (Univ of Sussex) |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Study participants or study members |
Results and Impact | A workshop intended to bring researchers with a common interest in the broad area of vision for movement together to describe methods specific to their fields which could be relevant to other areas of research. Outcomes included new collaborations, shared knowledge, and new funding avenues. |
Year(s) Of Engagement Activity | 2015 |
Description | Public lecture at Macquarie University |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | This was a flagship public lecture as part of a series of four organised by Faculty of Science and Engineering at Macquarie University (Australia) with an audience of over 100 people. |
Year(s) Of Engagement Activity | 2017 |
URL | http://www.mq.edu.au/about/events/view/public-lecture-navigation-in-insects-and-robots/ |
Description | Scotsman article |
Form Of Engagement Activity | A press release, press conference or response to a media enquiry/interview |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Media (as a channel to the public) |
Results and Impact | Invited to write Opinion column for the Scotsman newspaper |
Year(s) Of Engagement Activity | 2022 |
URL | https://www.scotsman.com/news/opinion/columnists/how-engineers-are-learning-from-the-powerful-circui... |
Description | Virtual Insect Navigation Workshop Aug 4th-6th |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Presentation at the Virtual Insect Navigation Workshop Aug 4th-6th |
Year(s) Of Engagement Activity | 2020 |
Description | Virtual ant experience |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Public/other audiences |
Results and Impact | Using a Oculus Rift headset, we produced a virtual reality game in which the participant experienced the ants point of view in attempting to navigate through the real environment of the ant (a virtual world created from a data set of laser scans in the ant's habitat). This was used at open days and local science festivals. |
Year(s) Of Engagement Activity | 2016 |