Multi-Robot Manipulation Planning for Forceful Manufacturing Tasks

Lead Research Organisation: University of Leeds
Department Name: Sch of Computing


Imagine coming back home from a hardware store with planks of wood and working with your friend to manufacture a table for yourself. You will need to collaborate to perform operations such as cutting parts off, inserting nails, drilling holes, and screwing in fasteners. The goal of this project is to get robots to perform similar manufacturing tasks. To do this, a robot team will need to decide how to grasp the workpieces (e.g. wood planks) and how to move to perform these operations. Within this project I will develop planning algorithms that will make such decisions.

When the planning algorithms make these decisions, they will need to solve several problems.

First, the algorithms must solve geometric problems. Suppose you want to hold a wooden panel such that your friend will drill a hole on a particular surface of it. You would hold the panel such that the surface your friend needs to drill is looking away from you (as opposed to holding the panel such that the surface is looking towards you), so that your friend can position her body across from you and drill comfortably. As you do this you are solving a geometric problem: you grasp the workpiece and position your body such that your collaborator will have the necessary space to position her own body and reach the workpiece to perform the operation. Collaborating robots must solve the same problem: they must grasp workpieces and position themselves such that they can all reach the workpiece and perform operations on it without colliding into each other. As the number of robots required to perform an operation increases, solving the geometric problem becomes harder and harder.

Second, the algorithms must solve stability problems due to forces applied on the workpiece. Keeping with the example above, you will probably hold the wooden panel such that you will rest your palms firmly against it close to the point of drilling to be able to resist the forces arising during the operation. This, first of all, requires predicting the direction and magnitude of the forces that will arise before the operation even starts. It also requires planning combinations of contact points on the workpiece and the configurations of your arms, such that you will be able to resist these forces. A robotic planner must solve these problems as well. A particular challenge is solving them simultaneously with the geometric problems mentioned above.

Finally, the algorithms must also solve sequential problems because manufacturing a complete product takes much more than a single operation. Rather the robots will need to perform multiple sequential operations such as drilling multiple holes one after the other, then cutting a piece off, and then inserting fasteners. A planner can naively choose to solve each operation separately. However, this would mean that every operation has its own set of grasps and geometric position, planned independently from each other. This would require un-grasping the workpiece after every single operation, and re-grasping and moving it for the next operation. On the other hand, if the algorithms can plan with foresight, e.g. if they can find grasp configurations which obey the geometric and stability constraints not only for the next immediate operation but also for the operations following it, then the final plan would be much more efficient to execute and the robots can avoid the redundant un-grasp/re-grasp operations. Planning multiple operations simultaneously, however, makes the problem even harder because the geometric and stability constraints of future operations must be considered earlier during planning.

The primary goal of this project is to develop a planning framework solving all these constraints. The proposed work also involves building a multi-robot system to test our algorithms. This multi-robot system will start with a pile of manufacturing materials and perform operations such as cutting, drilling, and fastening to build products.

Planned Impact

As the world population grows and nations develop, one of the economic and social challenges we face is the advancement of manufacturing to satisfy the increased and varied demand. Autonomous intelligent robots have the potential to address this challenge, by creating a step change in the way manufacturing is done. The most important impact of this research proposal will be to accelerate the progress in this direction.

Manufacturers which are small and medium-sized enterprises (SMEs) will benefit most from the intelligent planning technologies proposed here. In today's factory automation, robots are tediously programmed for every single action. This makes robots feasible only in high-volume manufacturing, where they are programmed once to manufacture thousands of copies of the same product.
According to the International Federation of Robotics [1] more than 75% of all current industrial robots are used by large manufacturers performing mass production. The two main sectors are automotive, which use about 50% of all industrial robots, and mass-produced electronics (smart phones and tablets) which use about 25% of all industrial robots. SMEs, even though they contribute 47% of the turnover of UK's private sector [2], make little use of robotic technology. These manufacturers include metals, wood, plastics, and machinery sectors with tasks such as cutting, drilling, grinding, milling, and polishing of products made of metals, wood, plastics and others. The high-variety/low-volume nature of SME manufacturing makes it infeasible to install and re-program new robotic systems every time a product or task is changed. This makes flexible manufacturing, i.e. the ability to adapt to new products/tasks easily, a desired feature in automation.

Intelligent planning algorithms, by enabling robots to adapt to new tasks without the need for re-programming, has the potential to make a major contribution to flexible automation. By enabling flexible robotic automation, intelligent manipulation planning can contribute to improved productivity, reduced costs, and competitiveness for SMEs. We will organize activities within this project to make the planners we develop visible to its potential beneficiaries; manufacturers in the metals, wood, plastics, and machinery sectors. Specifically, I propose to:
* Create an Interactive Robot Demonstration, which will use our robot set-up to show the planner's capabilities to manufacturers in an interactive way;
* Organize an Industry Day, which will bring manufacturers to our lab to learn about our system and to give feedback;
* Create ROS-Industrial Software Packages, which will provide potential users with an easy way to try our algorithms on their systems.

The details of these activities are presented in "Pathways to Impact".

Description We have developed an algorithm for a robot to plan grasping an object such that the object can resist forceful operations applied on it, such as drilling and cutting. A video can be seen here:
We have also extended this algorithm such that it can be a human performing these forceful operations, and the algorithm considers the comfort of the human during the activity.
Exploitation Route With further improvement, such a system can be used in manufacturing/assembly environments, where a workpiece undergoes multiple operations, such as drilling, cutting, milling, polishing, etc.
Sectors Aerospace, Defence and Marine,Electronics,Manufacturing, including Industrial Biotechology

Description Findings of this project led to discussions with companies in the industry (companies including Amazon Robotics, Cavendish Nuclear, Advanced Supply Chain Group, Zebra Technologies, Bosch Research). These discussions led to new proposed projects on using robots in manufacturing and warehouses. One such project was funded, with support from Amazon Robotics and Advanced Supply Chain Group. This project is currently active, and the goal is to make the UK competitive in robotic automation of warehouse picking and packing. This is an area with increasing activity in many leading countries, and the outputs of this project help the UK to stay at the cutting edge of technology in automated warehouse picking and packing. Personnel from this project went on to be employed by leading Robotics companies (e.g., Tencent Robotics) to focus on projects related to robotic manipulation in manufacturing.
First Year Of Impact 2022
Sector Electronics,Manufacturing, including Industrial Biotechology
Impact Types Economic

Description Robotic picking and packing with physical reasoning
Amount £1,196,800 (GBP)
Funding ID EP/V052659/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Public
Country United Kingdom
Start 12/2021 
End 11/2026
Description Planning for human comfort during human-robot collaborative manipulation 
Organisation University of Leeds
Country United Kingdom 
Sector Academic/University 
PI Contribution This was a collaboration with Dr Samit Chakrabarty from School of Biomedical Engineering, University of Leeds. We extended our robotic planners to include musculoskeletal definitions of human comfort, so that robots can choose actions that are comfortable for humans (e.g. where to hold an object a robot and a human are jointly manipulating).
Collaborator Contribution They helped us with definitions of human comfort and experiments to measure human muscle activity via EMG during human-robot collaboration.
Impact This collaboration led to a publication at the top-ranked journal in human-robot interaction (ACM Transactions in HRI), which then was invited for an oral presentation also at the top-ranked ACM/IEEE International Conference on Human-Robot Interaction, 2023, in Stockholm. We are also in the process of writing a follow-up joint grant proposal with Dr Chakrabarty.
Start Year 2018
Description 3rd UK Robot Manipulation Workshop 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Other audiences
Results and Impact We organized the 3rd UK Robot Manipulation Workshop in Leeds, with attendance from more than 100 researchers and industrial practitioners from all around the UK. The two-day event included talks by leading researchers in the field, and poster presentations from postgraduate researchers.
The event website can be found here:
Year(s) Of Engagement Activity 2019
Description Industry visits 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Industry/Business
Results and Impact We had companies, including Unilever, Rakusens Crackers, and others to see our robotic manipulation research to get informed about current technology on the potential of intelligent robotics in manufacturing.
Year(s) Of Engagement Activity 2018
Description Open Days for Undergrad Applicants and Parents 
Form Of Engagement Activity Participation in an open day or visit at my research institution
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Schools
Results and Impact We had multiple such Open Days / Applicant Days where we demonstrated our research. This has increased interest in our School and our Robotics activity.
Year(s) Of Engagement Activity 2018