ALOOF: Autonomous Learning of the Meaning of Objects
Lead Research Organisation:
University of Birmingham
Department Name: School of Computer Science
Abstract
When working with and for humans, robots and autonomous systems must know about the objects involved in human activities, e.g. the parts and tools in manufacturing, the professional items used in service applications, and the objects of daily life in assisted living. While great progress has been made in object instance and class recognition, a robot is always limited to knowing about the objects it has been trained to recognize. The goal of ALOOF is to enable robots to exploit the vast amount of knowledge on the Web in order to learn about previously unseen objects and to use this knowledge when acting in the real world. We will develop techniques to allow robots to use the Web to not just learn the appearance of new objects, but also their properties including where they might be found in the robot's environment.
To achieve our goal, we will provide a mechanism for translating between the representations robots use in their real-world experience and those found on the Web. Our proposed translation mechanism is a meta-modal representation (i.e. a representation which contains and structures representations from other modalities), composed of meta-modal entities and relations between them. A single entity represents a single object type, and is composed of modal features extracted from robot sensors or the Web. The combined features are linked to the semantic properties associated with each entity. The robot's collection of meta-modal entities is organized into a structured ontology, supporting formal reasoning. This representation is complemented with methods for detecting gaps in the knowledge of the robot (i.e. unknown objects and properties), and for planning how to fill these gaps. As the robot's main source of new knowledge will be the Web, we will also contribute techniques for extracting relevant knowledge from Web resources using novel machine reading and computer vision algorithms.
By linking meta-modal representations with the perception and action capabilities of robots, we will achieve an innovative and powerful mix of Web-supported and physically-grounded life-long learning. Our scenario consists of an open-ended domestic setting where robots have to find objects. Our measure of progress will be how many knowledge gaps (i.e. situations where the robot has incomplete information about objects), can be resolved autonomously given specific prior knowledge. We will integrate the results on multiple mobile robots including the MetraLabs SCITOS robot, and the home service robot HOBBIT.
To achieve our goal, we will provide a mechanism for translating between the representations robots use in their real-world experience and those found on the Web. Our proposed translation mechanism is a meta-modal representation (i.e. a representation which contains and structures representations from other modalities), composed of meta-modal entities and relations between them. A single entity represents a single object type, and is composed of modal features extracted from robot sensors or the Web. The combined features are linked to the semantic properties associated with each entity. The robot's collection of meta-modal entities is organized into a structured ontology, supporting formal reasoning. This representation is complemented with methods for detecting gaps in the knowledge of the robot (i.e. unknown objects and properties), and for planning how to fill these gaps. As the robot's main source of new knowledge will be the Web, we will also contribute techniques for extracting relevant knowledge from Web resources using novel machine reading and computer vision algorithms.
By linking meta-modal representations with the perception and action capabilities of robots, we will achieve an innovative and powerful mix of Web-supported and physically-grounded life-long learning. Our scenario consists of an open-ended domestic setting where robots have to find objects. Our measure of progress will be how many knowledge gaps (i.e. situations where the robot has incomplete information about objects), can be resolved autonomously given specific prior knowledge. We will integrate the results on multiple mobile robots including the MetraLabs SCITOS robot, and the home service robot HOBBIT.
Planned Impact
ALOOF will change how we work with robots. Today, it is hardly believable that a robot could provide you with information about any object you show to it. After ALOOF we will be able to ask future robots to (1) find an object given a name, (2) find out the name for an unknown object and (3) find out other information about an object. ALOOF technology will enable any Web-connected device (e.g. from smart phones to robots) to learn from the Web, widening the range of applications that can profit from autonomous online knowledge gathering. This could range from driver assistance systems that could learn about locally unique buildings, signs, or sites, to Ambient Assistive Living (AAL) systems. In general, the capability to learn from the Web will become an essential tool for technology that must perform some task in loosely-structured and open-ended environments, thus creating impact (in terms of increased productivity and capability) in a wide range of fields, from supporting independent living to creating new shopping experiences.
In order to encourage the use of robots in new service scenarios, and to support the predicted growth in the service robot market, it is crucial that future service robots are autonomous and robust, and also as cheap and easy to deploy as possible. This cannot be said for current technology, as robots must be trained in advance for all objects they must work with (impossible in open-ended domains, as found in most service scenarios). ALOOF will change this. By using Web learning integrated with situated experience, robots can train themselves to recognize objects as they encounter them, thus minimising deployment overhead. By being founded on an approach that assumes missing knowledge is always part of a problem, ALOOF systems will be uniquely robust to failures in perception, using Web learning to fill in the gaps in their understanding. Successful demonstrations of these technologies will increase the uptake of robots based on UK technology in both targeted industries and beyond, increasing market share and UK competitiveness in the process.
In order to encourage the use of robots in new service scenarios, and to support the predicted growth in the service robot market, it is crucial that future service robots are autonomous and robust, and also as cheap and easy to deploy as possible. This cannot be said for current technology, as robots must be trained in advance for all objects they must work with (impossible in open-ended domains, as found in most service scenarios). ALOOF will change this. By using Web learning integrated with situated experience, robots can train themselves to recognize objects as they encounter them, thus minimising deployment overhead. By being founded on an approach that assumes missing knowledge is always part of a problem, ALOOF systems will be uniquely robust to failures in perception, using Web learning to fill in the gaps in their understanding. Successful demonstrations of these technologies will increase the uptake of robots based on UK technology in both targeted industries and beyond, increasing market share and UK competitiveness in the process.
Organisations
Publications
Faulhammer T
(2017)
Autonomous Learning of Object Models on a Mobile Robot
in IEEE Robotics and Automation Letters
Kunze L
(2018)
IEEE Robotics and Automated Letters
in Artificial Intelligence for Long-Term Robot Autonomy: A Survey
Massouh Nizar
(2017)
Learning Deep Visual Object Models From Noisy Web Data: How to Make it Work
in arXiv e-prints
Description | BBC Radio 4's The Now Show |
Form Of Engagement Activity | A broadcast e.g. TV/radio/film/podcast (other than news/press) |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | Interview on the BBC Radio 4's The Now Show about DeepMind and AlphaGo. |
Year(s) Of Engagement Activity | 2016 |
URL | http://www.bbc.co.uk/programmes/b072n8fq |
Description | Business Magazine Tech 100 after dinner talk |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Regional |
Primary Audience | Industry/Business |
Results and Impact | Talk to the Business Magazine Tech 100 launch event about autonomous systems and AI. |
Year(s) Of Engagement Activity | 2016 |
URL | http://www.businessmag.co.uk/southerntech100/southerntech100southern-tech-dinner-2016/ |
Description | Harvey Nash Technology Survey launch event |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Industry/Business |
Results and Impact | A talk on autonomous robots and AI at Harvey Nash Technology Survey launch event to tech industry CTOs and recruiters. |
Year(s) Of Engagement Activity | 2015 |
URL | http://www.harveynash.com/uk/news-and-media/practices-technology-news/harvey_nash_technology_survey_... |
Description | Industry Talk, Hive |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Industry/Business |
Results and Impact | Industry talk, HIVE, Kensington London |
Year(s) Of Engagement Activity | 2017 |
URL | http://thehive-network.com/wp-content/uploads/2017/11/Retail-Hive-Innovation-Final-Report.pdf |
Description | Interview for Factor Magazine |
Form Of Engagement Activity | A press release, press conference or response to a media enquiry/interview |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | Interview for Factor magazine about Facebook founder Mark Zuckerberg building an AI-controlled butler. |
Year(s) Of Engagement Activity | 2016 |
URL | http://factor-tech.com/feature/zuckerberg-and-the-robot-butler/ |
Description | Level Up Human podcast |
Form Of Engagement Activity | A broadcast e.g. TV/radio/film/podcast (other than news/press) |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | Recording two episodes of the Level Up Human podcast, covering topics on AI and robotics. |
Year(s) Of Engagement Activity | 2016 |
URL | http://leveluphuman.com/ |
Description | Plenary at Taros 2016 |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | Plenary at Taros 2016 |
Year(s) Of Engagement Activity | 2016 |
Description | Proctor & Gamble: Outside In |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Regional |
Primary Audience | Industry/Business |
Results and Impact | A talk on AI/Robotics for on of Proctor & Gamble's technology teams as part of their "Outside In" series of talks bringing in exciting external ideas. |
Year(s) Of Engagement Activity | 2016 |
Description | Public engagement talk, 6th December 2017 |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Public/other audiences |
Results and Impact | Public engagement at Derby Institute of Physics Christmas Lecture, 6th December 2017 |
Year(s) Of Engagement Activity | 2017 |
Description | Public engagement, 25th April 2017 |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Public/other audiences |
Results and Impact | Talk at Warwickshire WI, 25th April 2017 |
Year(s) Of Engagement Activity | 2017 |
Description | Research Talk, 21st February 2017 |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | Talk at IEEE UK and Ireland Robotics and Automation Symposium |
Year(s) Of Engagement Activity | 2017 |
URL | https://www.ieee-ukandireland.org/event/the-first-annual-ieee-uk-ireland-robotics-and-automation-sym... |
Description | Robotics Innovation Show |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Industry/Business |
Results and Impact | Gave a talk on autonomous mobile service robots to the Robotics Innovation Show, a trade show for robots and autonomous systems. The talk led to some new relationships with partners interested in deploying autonomous service robots. |
Year(s) Of Engagement Activity | 2016 |
URL | http://www.roboticstoday.com/events/robotics-innovation-show |
Description | Talk - Birmingham, 8th June 2016 |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Public/other audiences |
Results and Impact | Talk was called: "I, for one, welcome out new robot overlords at Skeptics in the pub" |
Year(s) Of Engagement Activity | 2016 |
Description | Talk at University of the Third Age annual conference, 24th August 2016 |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Regional |
Primary Audience | Postgraduate students |
Results and Impact | Talk at University of Third Age annual conference |
Year(s) Of Engagement Activity | 2016 |
Description | Talk at York Science Festival, 10th June 2016 |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | Talk at York Science Festival, discussed how artificial intelligence is being used to make robots more independent, and where these smart robots will be used in the near future. Hear about the latest research to make robots plan, learn, see, communicate and navigate. What are the big challenges for the future of robotics and how close are we to over-coming them? |
Year(s) Of Engagement Activity | 2016 |
URL | http://yorkfestivalofideas.com/2016/talks/inside-future-robots/ |
Description | Talk, Barnsley, 6th July 2017 |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Public/other audiences |
Results and Impact | Talk was called: "I, for one, welcome out new robot overlords at Skeptics in the pub" |
Year(s) Of Engagement Activity | 2017 |
Description | Talk, Brighton, 1st December 2-16,7pm |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Public/other audiences |
Results and Impact | Talk was called: "I, for one, welcome out new robot overlords at Skeptics in the pub" |
Year(s) Of Engagement Activity | 2016 |
Description | Talk, Leicester, 20th June 2017 |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Public/other audiences |
Results and Impact | Talk was called: "I, for one, welcome out new robot overlords at Skeptics in the pub" |
Year(s) Of Engagement Activity | 2017 |
Description | Talk, Liverpool, 3rd May 2016 |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Public/other audiences |
Results and Impact | Long-Term Autonomy in Everyday Environments: A New Challenge for AI and Robotics 3rd May 2016, 13:00, Ashton Lecture Theater Dr Nick Hawes School of Computer Science University of Birmingham Edgbaston, Birmingham, B15 2TT United Kingdom Abstract The performance of autonomous robots, i.e. robots that can make their own decisions and choose their own actions, is becoming increasingly impressive, but most of them are still constrained to labs, or controlled environments. In addition to this, these robots are typically only able to do intelligent things for a short period of time, before either crashing (physically or digitally) or running out of things to do. In order to go beyond these limitations, and to deliver the kind of autonomous service robots required by society, we must conquer the challenge of combining artificial intelligence and robotics to develop systems capable of long-term autonomy in everyday environments. This talk will present an overview of research in this direction, focussing on the mobile robots for security and care domains developed by the EU-funded STRANDS project (http://strands-project.eu). |
Year(s) Of Engagement Activity | 2016 |
URL | https://intranet.csc.liv.ac.uk/research/seminars/abstract2.php?id=189 |
Description | Talk, Nottingham, 6th February 2018 |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Public/other audiences |
Results and Impact | Talk was called: "I, for one, welcome out new robot overlords at Skeptics in the pub" |
Year(s) Of Engagement Activity | 2018 |
Description | Talk, Stoke, 5th April 2017, 17:15pm |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Public/other audiences |
Results and Impact | Talk was called: "I, for one, welcome out new robot overlords at Skeptics in the pub" |
Year(s) Of Engagement Activity | 2017 |
Description | Tutorial - 2017 |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Undergraduate students |
Results and Impact | * The Fourth Lisbon Summer School on "Articial Intelligence and Robotics" - 2017 - |
Year(s) Of Engagement Activity | 2017 |
URL | http://lucia.isr.tecnico.ulisboa.pt |
Description | Tutorial 2016 |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Undergraduate students |
Results and Impact | The Third O¨rebro Winter School on "Articial Intelligence and Robotics" - 2016 - |
Year(s) Of Engagement Activity | 2016 |
URL | http://aass.oru.se/Agora/Lucia2016/ |
Description | Tutorial, ICAPS 2016 Summer School |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Undergraduate students |
Results and Impact | Tutorial at summer school 2016, ICAPS |
Year(s) Of Engagement Activity | 2016 |
URL | http://icaps16.icaps-conference.org/summer-school.html |
Description | Tutorial, October 2016 |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Undergraduate students |
Results and Impact | http://wasp-sweden.org/graduate-school/courses/autonomous-systems/learning-and-knowledge-oct-27-28-stockholm/ |
Year(s) Of Engagement Activity | 2016 |
URL | http://wasp-sweden.org/graduate-school/courses/autonomous-systems/learning-and-knowledge-oct-27-28-s... |
Description | University of the Third Age at the Royal Institution |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Public/other audiences |
Results and Impact | A talk on autonomous robots to the University of the Third Age at the Royal Institution. My main topic was the (lack of an) existential threat from future AI and robots. |
Year(s) Of Engagement Activity | 2015 |
URL | http://www.rigb.org/whats-on/events-2015/october/public-u3a-at-the-ri-relativity-reproduction-and-ro... |
Description | Workshop, 10th March 2017 |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Public/other audiences |
Results and Impact | EPSRC V&V Network Workshop on Verification and Validation of Sensing and Control Models in Autonomous Systems. |
Year(s) Of Engagement Activity | 2017 |
URL | https://vavas.org/previousevents/verification-and-validation-of-sensing-and-control-models-in-autono... |