HEAP: Human-Guided Learning and Benchmarking of Robotic Heap Sorting

Lead Research Organisation: University of Lincoln
Department Name: School of Computer Science

Abstract

This project will provide scientific advancements for benchmarking, object recognition, manipulation and human-robot interaction. We focus on sorting a complex, unstructured heap of unknown objects --resembling nuclear waste consisting of a set of broken deformed bodies-- as an instance of an extremely complex manipulation task. The consortium aims at building an end-to-end benchmarking framework, which includes rigorous scientific methodology and experimental tools for application in realistic scenarios.
Benchmark scenarios will be developed with off-the-shelf manipulators and grippers, allowing to create an affordable setup that can be easily reproduced both physically and in simulation. We will develop benchmark scenarios with varying complexities, i.e., grasping and pushing irregular objects, grasping selected objects from the heap, identifying all object instances and sorting the objects by placing them into corresponding bins. We will provide scanned CAD models of the objects that can be used for 3D printing in order to recreate our benchmark scenarios. Benchmarks with existing grasp planners and manipulation algorithms will be implemented as baseline controllers that are easily exchangeable using ROS.

The ability of robots to fully autonomously handle dense clutters or a heap of unknown objects has been very \textit{limited} due to challenges in scene understanding, grasping, and decision making. Instead, we will rely on semi-autonomous approaches where a human operator can interact with the system (e.g. using tele-operation but not only) and giving high-level commands to complement the autonomous skill execution. The amount of autonomy of our system will be adapted to the complexity of the situation. We will also benchmark our semi-autonomous task execution
with different human operators and quantify the gap to the current SOTA in autonomous manipulation. Building on our semi-autonomous control framework, we will develop a manipulation skill learning system that learns from demonstrations and corrections of the human operator and can therefore learn complex manipulations in a data-efficient manner. To improve object recognition and segmentation in cluttered heaps, we will develop new perception algorithms and investigate interactive perception in order to improve the robot's understanding of the scene in terms of object instances, categories and properties.

Planned Impact

nA

Publications

10 25 50
 
Description HEAP is a research project funded by Chist-Era that investigates Robot Manipulation Algorithms for Robotic Heap Sorting. This project will provide scientific advancements for benchmarking, object recognition, manipulation and human-robot interaction. We focus on sorting a complex, unstructured heap of unknown objects -resembling nuclear waste consisting of a set of broken deformed bodies- as an instance of an extremely complex manipulation task. The consortium aims at building an end-to-end benchmarking framework, which includes rigorous scientific methodology and experimental tools for application in realistic scenarios. Benchmark scenarios are being developed with off-the-shelf manipulators and grippers, allowing to create an affordable setup that can be easily reproduced both physically and in simulation. We are developing benchmark scenarios with varying complexities, i.e., grasping and pushing irregular objects, grasping selected objects from the heap, identifying all object instances and sorting the objects by placing them into corresponding bins. We will provide scanned CAD models of the objects that can be used for 3D printing in order to recreate our benchmark scenarios. Benchmarks with existing grasp planners and manipulation algorithms will be implemented as baseline controllers that are easily exchangeable using ROS. We developed benchmarking guidelines for robot grasping that are also integrated in our simulation environment.

The ability of robots to fully autonomously handle dense clutters or a heap of unknown objects has been very limited due to challenges in scene understanding, grasping, and decision making. Instead, we are relying on semi-autonomous approaches where a human operator can interact with the system (e.g. using tele-operation but not only) and giving high-level commands to complement the autonomous skill execution. The amount of autonomy of our system will be adapted to the complexity of the situation. First human-user studies with our tele-operation system have already been performed. We will also benchmark our semi-autonomous task execution with different human operators and quantify the gap to the current SOTA in autonomous manipulation. Building on our semi-autonomous control framework, we are developing a manipulation skill learning system that learns from demonstrations and corrections of the human operator and can therefore learn complex manipulations in a data-efficient manner. To improve object recognition and segmentation in cluttered heaps, we are developing new perception algorithms and investigate interactive perception in order to improve the robot's understanding of the scene in terms of object instances, categories and properties.

The activity in the first year concentrated on creating benchmarking testbeds for robot heap sorting in simulation as well as on the real robot. We additionally developped new intutive tele-operation approaches and conducted preliminary user studies with it for controlling the robot and simple manipulations of heaps. This tele-operation approach will be used for shared control experiments as well as for collecting a lot of demonstration data. Other European partners worked on benchmarking protocols for robot grasping, probabilistic scene segmentation algorithms and user-guided learning of new grasp strategies. The UoL team is now concentrating on active segmentation algorithms, i.e., how the robot can actively improve the segmentation of the scene (e.g. acquire by mask RCNN) by moving/pushing objects
Exploitation Route Robot manipulation algorithms for heap sorting are very important in manufacturing as well as for waste decommissioning. The HEAP project has strong ties with other projects like the National Center for Nuclear Robotics (NCNR), where similar algorithms are used for waste decommissioning. We also started a collaboration with Bosch BCAI in Renningen (Germany), who are interested in these algorithms for Bin Picking applications.
Sectors Manufacturing, including Industrial Biotechology

URL https://heap-chist-era.github.io/about/
 
Description Bosch 
Organisation Bosch Group
Country Global 
Sector Private 
PI Contribution We defined the benchmarking scenarios that Bosch (BCAI in Renningen) is also using to evaluate their algorithms.
Collaborator Contribution Bosch is also interested in the Benchmarking scenarios and actively contributing the the project with a small research group.
Impact - Addional Benchmarks for heap manipulation
Start Year 2019
 
Description HEAP Consortium 
Organisation Idiap Research Institute
Country Switzerland 
Sector Charity/Non Profit 
PI Contribution The Heap project consists of an international consortium funded by chist-Era. Our project focuses on benchmarking robotic manipulation algorithms for sorting a heap of unknown objects. The contributions from the University of Lincoln were to develop the specifications of the robot benchmark platform as well as the simulation framework for benchmarking heap manipulations. We are actively collaborating with IDIAP on learning pushing and grasping strategies and with TU Vienna on active scene segmentation.
Collaborator Contribution Other partners are implementing benchmark scenarios and benchmarking state of the art algorithms. Moreover, new algorithms for grasping, scene segmentation and pushing are being developed by our partners.
Impact - Definition and implementation of benchmarking scenarios for heap manipulation
Start Year 2019
 
Description HEAP Consortium 
Organisation Italian Institute of Technology (Istituto Italiano di Tecnologia IIT)
Country Italy 
Sector Academic/University 
PI Contribution The Heap project consists of an international consortium funded by chist-Era. Our project focuses on benchmarking robotic manipulation algorithms for sorting a heap of unknown objects. The contributions from the University of Lincoln were to develop the specifications of the robot benchmark platform as well as the simulation framework for benchmarking heap manipulations. We are actively collaborating with IDIAP on learning pushing and grasping strategies and with TU Vienna on active scene segmentation.
Collaborator Contribution Other partners are implementing benchmark scenarios and benchmarking state of the art algorithms. Moreover, new algorithms for grasping, scene segmentation and pushing are being developed by our partners.
Impact - Definition and implementation of benchmarking scenarios for heap manipulation
Start Year 2019
 
Description HEAP Consortium 
Organisation Vienna University of Technology
Country Austria 
Sector Academic/University 
PI Contribution The Heap project consists of an international consortium funded by chist-Era. Our project focuses on benchmarking robotic manipulation algorithms for sorting a heap of unknown objects. The contributions from the University of Lincoln were to develop the specifications of the robot benchmark platform as well as the simulation framework for benchmarking heap manipulations. We are actively collaborating with IDIAP on learning pushing and grasping strategies and with TU Vienna on active scene segmentation.
Collaborator Contribution Other partners are implementing benchmark scenarios and benchmarking state of the art algorithms. Moreover, new algorithms for grasping, scene segmentation and pushing are being developed by our partners.
Impact - Definition and implementation of benchmarking scenarios for heap manipulation
Start Year 2019
 
Description Lecture on Policy Gradients Methods at the RLSS Summer school in Lille, July 2019 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact I was invited to give a 2 hour lecture on Robot Reinforcement Learning and Policy Search at the Reinforcement Learning Summer School in Lille 2019. There were about 150 participants (mostly PhD Students) attending the lecture.
Year(s) Of Engagement Activity 2019
URL https://rlss.inria.fr/
 
Description National Nuclear User Facility for Hot Robotics (NNUF-HR) Town Hall Meeting 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Academic and industry stakeholders attended the meeting. The day included presentations from NNUF regional nodes, where participants discussed ideas ideas and recommendations for equipment and capabilities that the facility will offer.
Year(s) Of Engagement Activity 2019
URL https://southwestnuclearhub.ac.uk/event/national-nuclear-user-facility-hot-robotics-town-hall-meetin...
 
Description Presentation at RACE 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact 10 researchers from UKAEA attended the talk, which sparked discussion on potential venues of collaboration and the use of teleoperation for nuclear handling withing the RACE premises. The company reported interest in the subject area and future collaboration plans are drawn.
Year(s) Of Engagement Activity 2019
 
Description Workshop organization at RSS 2019 - Task Informed Robot Grasping 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact Together with colleagues I organized the 2nd Task-Informed Grasping workshop at the RSS conference. There were invited talks and a poster session. About 120 people attended the event.
Year(s) Of Engagement Activity 2019
URL https://lcas.lincoln.ac.uk/wp/tig-ii/