Robotic picking and packing with physical reasoning

Lead Research Organisation: University of Leeds
Department Name: Sch of Computing


This Fellowship focuses on robotic object manipulation. Object manipulation refers to all the different ways robots can interact with objects in their environments. Consider the example of packing different items into a box in a warehouse to ship the box to a customer. To perform this, a robot would need to pick and insert the items into the box one by one, while nudging, pushing and squeezing objects, to achieve a tight packing. The robot would need to plan and control its actions, as well as use sensors to estimate the positions and deformations of the objects in the box.

The dominant approach to object manipulation in the literature is geometry-based, where the world is represented using shapes and configurations only. While this simplifies planning and control, it also results in robots that are extremely limited in their skills. The central vision of this Fellowship is to go beyond that, by enabling robots to reason about, plan in, and control the full physics of the world. This has the potential to transform robots' object manipulation skills and our lives, because robots will be able to perform a much diverse variety of object manipulation skills applicable to manufacturing, assembly, and services.

This Fellowship will create fundamental algorithms --- algorithms that can be applied to different object manipulation problems by other researchers and engineers. However, this Fellowship will also target a particular application area: picking and packing of objects for warehouse automation. With the rapid advance of e-commerce over the past decade, there is a pressing need to have efficient warehouse automation systems, in the UK and the world. However, existing robotic systems do not have physics-based reasoning, which limits their applications drastically. The physics-based picking and packing approach that I propose will enable robots to reach into cluttered bins/shelves/bags, pushing, nudging, and squeezing arbitrary objects to search and retrieve a particular object or to pack multiple objects tightly for shipping --- skills that do not exist in any current system.

There are significant challenges to using physics-based models during robotic manipulation. An important one is computational expense. We have low-level physics models and physics engines, similar to the ones used in computer games, which can be used by robots. However, computing such models are expensive (i.e. takes too much computer time) and robotic algorithms need to query such models thousands, and sometimes millions, of times before choosing an action, making it infeasible to use such a straightforward approach. Instead, I propose to develop and use hierarchical models of physics. At higher levels in this hierarchy are coarse, approximate physics models, i.e. models that are computationally cheap (i.e. fast to compute) but may be inaccurate. At lower levels in this hierarchy are fine models, i.e. models that are computationally expensive (i.e. slow to compute) but are accurate. I will investigate a variety of methods (including data-driven methods as well as parallel computing methods) to learn and compute such a hierarchy of physics models. I will also develop new planning, control, and state estimation algorithms that can use these new hierarchical physics models.

I will also use these new algorithms and systems to accelerate the adoption of this technology in the UK. I will work with the EPSRC UK Robotics & Autonomous Systems Network and industrial stakeholders to develop a roadmap for the integration of autonomous picking and packing robots into the existing industrial workflows. I will also aim to create a new national organisation to focus on this important technology.

To achieve these aims, I will work with many academic and industrial partners, including the Advanced Supply Chain Group, a leading UK-based supply chain and warehouse management company, as well as a key international company in this area, Amazon.
Description We have been developing methods which show that physics-based reasoning help robots track objects in cluttered environments. A video demonstration can be seen here:
We have also released our code related to this here:
Exploitation Route These outcomes can enable robots to manipulate objects in cluttered environments, for example in warehouse picking/packing applications, or in people's homes, e.g., when reaching into a cabinet or fridge shelf.
Sectors Agriculture

Food and Drink


Description Multi-object manipulation (with UC Berkeley) 
Organisation University of California, Berkeley
Country United States 
Sector Academic/University 
PI Contribution This is a collaboration between our group at University of Leeds and Prof. Ken Goldberg's group at UC Berkeley, USA, who are also partners of my EPSRC Fellowship. We have contributed to the development of methods for the manipulation and grasping of mutiple, rigid and deformable, objects by robots. We developed an initial prototype of the multi-object grasping system in Leeds. The system was further developed at UC Berkeley, together with a visitor from our team, and was recently generalized to deformable objects (e.g., garments/clothes). We also contributed with help to the writing of publications.
Collaborator Contribution The UC Berkeley team developed the core system as well as the algorithms methods, in collaboration with our team. They have performed most of the experimental work in their labs. They also led the writing of the publications.
Impact The collaboration resulted in four publications so far.
Start Year 2021
Description Physics-based model-predictive control for object manipulation (with ETH Zurich) 
Organisation ETH Zurich
Country Switzerland 
Sector Academic/University 
PI Contribution I visited Prof. Robert Katzschmann's group in ETH Zurich, who are also partners on my EPSRC Fellowship. During the visit, I learned about the tendon-driven, anthropomorphic robotic hand developed by Prof. Katzschmann's group, as well as the different learning/AI methods they are exploring to perform autonomous manipulation with this hand. I have contributed with inputs to the developments of such methods, particularly leading a method based on physics-based model-predictive control. This work is still ongoing.
Collaborator Contribution Prof. Katzschmann's group contributed by providing the robotic hand hardware, as well as all other necessary tools for the development.
Impact This collaboration does not have any outputs yet. It is ongoing work.
Start Year 2023
Description Use of tactile sensors in physics-based manipulation (with Univ. of Bristol) 
Organisation University of Bristol
Country United Kingdom 
Sector Academic/University 
PI Contribution Integrating tactile sensors developed by Prof. Nathan Lepora's group at University of Bristol (who are partners on my EPSRC Fellowship) into our physics-based object perception algorithms and methods.
Collaborator Contribution Prof. Nathan Lepora's group contributed by teaching us how to manufacture one of their tactip sensors, including an on-site training of the PDRA, and also help with the software that accompanies this sensor.
Impact Ongoing collaboration. No outputs yet.
Start Year 2023
Description Visit by ASDA Logistics Services 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Industry/Business
Results and Impact A visit was performed by Asda Logistics Services to our labs to discuss possible applications of our object manipulation methods, in Asda warehouses. This was then followed by a visit from our researchers to Asda warehouses.
Year(s) Of Engagement Activity 2023