Multi-Robot Manipulation Planning for Forceful Manufacturing Tasks
Lead Research Organisation:
University of Leeds
Department Name: Sch of Computing
Abstract
Imagine coming back home from a hardware store with planks of wood and working with your friend to manufacture a table for yourself. You will need to collaborate to perform operations such as cutting parts off, inserting nails, drilling holes, and screwing in fasteners. The goal of this project is to get robots to perform similar manufacturing tasks. To do this, a robot team will need to decide how to grasp the workpieces (e.g. wood planks) and how to move to perform these operations. Within this project I will develop planning algorithms that will make such decisions.
When the planning algorithms make these decisions, they will need to solve several problems.
First, the algorithms must solve geometric problems. Suppose you want to hold a wooden panel such that your friend will drill a hole on a particular surface of it. You would hold the panel such that the surface your friend needs to drill is looking away from you (as opposed to holding the panel such that the surface is looking towards you), so that your friend can position her body across from you and drill comfortably. As you do this you are solving a geometric problem: you grasp the workpiece and position your body such that your collaborator will have the necessary space to position her own body and reach the workpiece to perform the operation. Collaborating robots must solve the same problem: they must grasp workpieces and position themselves such that they can all reach the workpiece and perform operations on it without colliding into each other. As the number of robots required to perform an operation increases, solving the geometric problem becomes harder and harder.
Second, the algorithms must solve stability problems due to forces applied on the workpiece. Keeping with the example above, you will probably hold the wooden panel such that you will rest your palms firmly against it close to the point of drilling to be able to resist the forces arising during the operation. This, first of all, requires predicting the direction and magnitude of the forces that will arise before the operation even starts. It also requires planning combinations of contact points on the workpiece and the configurations of your arms, such that you will be able to resist these forces. A robotic planner must solve these problems as well. A particular challenge is solving them simultaneously with the geometric problems mentioned above.
Finally, the algorithms must also solve sequential problems because manufacturing a complete product takes much more than a single operation. Rather the robots will need to perform multiple sequential operations such as drilling multiple holes one after the other, then cutting a piece off, and then inserting fasteners. A planner can naively choose to solve each operation separately. However, this would mean that every operation has its own set of grasps and geometric position, planned independently from each other. This would require un-grasping the workpiece after every single operation, and re-grasping and moving it for the next operation. On the other hand, if the algorithms can plan with foresight, e.g. if they can find grasp configurations which obey the geometric and stability constraints not only for the next immediate operation but also for the operations following it, then the final plan would be much more efficient to execute and the robots can avoid the redundant un-grasp/re-grasp operations. Planning multiple operations simultaneously, however, makes the problem even harder because the geometric and stability constraints of future operations must be considered earlier during planning.
The primary goal of this project is to develop a planning framework solving all these constraints. The proposed work also involves building a multi-robot system to test our algorithms. This multi-robot system will start with a pile of manufacturing materials and perform operations such as cutting, drilling, and fastening to build products.
When the planning algorithms make these decisions, they will need to solve several problems.
First, the algorithms must solve geometric problems. Suppose you want to hold a wooden panel such that your friend will drill a hole on a particular surface of it. You would hold the panel such that the surface your friend needs to drill is looking away from you (as opposed to holding the panel such that the surface is looking towards you), so that your friend can position her body across from you and drill comfortably. As you do this you are solving a geometric problem: you grasp the workpiece and position your body such that your collaborator will have the necessary space to position her own body and reach the workpiece to perform the operation. Collaborating robots must solve the same problem: they must grasp workpieces and position themselves such that they can all reach the workpiece and perform operations on it without colliding into each other. As the number of robots required to perform an operation increases, solving the geometric problem becomes harder and harder.
Second, the algorithms must solve stability problems due to forces applied on the workpiece. Keeping with the example above, you will probably hold the wooden panel such that you will rest your palms firmly against it close to the point of drilling to be able to resist the forces arising during the operation. This, first of all, requires predicting the direction and magnitude of the forces that will arise before the operation even starts. It also requires planning combinations of contact points on the workpiece and the configurations of your arms, such that you will be able to resist these forces. A robotic planner must solve these problems as well. A particular challenge is solving them simultaneously with the geometric problems mentioned above.
Finally, the algorithms must also solve sequential problems because manufacturing a complete product takes much more than a single operation. Rather the robots will need to perform multiple sequential operations such as drilling multiple holes one after the other, then cutting a piece off, and then inserting fasteners. A planner can naively choose to solve each operation separately. However, this would mean that every operation has its own set of grasps and geometric position, planned independently from each other. This would require un-grasping the workpiece after every single operation, and re-grasping and moving it for the next operation. On the other hand, if the algorithms can plan with foresight, e.g. if they can find grasp configurations which obey the geometric and stability constraints not only for the next immediate operation but also for the operations following it, then the final plan would be much more efficient to execute and the robots can avoid the redundant un-grasp/re-grasp operations. Planning multiple operations simultaneously, however, makes the problem even harder because the geometric and stability constraints of future operations must be considered earlier during planning.
The primary goal of this project is to develop a planning framework solving all these constraints. The proposed work also involves building a multi-robot system to test our algorithms. This multi-robot system will start with a pile of manufacturing materials and perform operations such as cutting, drilling, and fastening to build products.
Planned Impact
As the world population grows and nations develop, one of the economic and social challenges we face is the advancement of manufacturing to satisfy the increased and varied demand. Autonomous intelligent robots have the potential to address this challenge, by creating a step change in the way manufacturing is done. The most important impact of this research proposal will be to accelerate the progress in this direction.
Manufacturers which are small and medium-sized enterprises (SMEs) will benefit most from the intelligent planning technologies proposed here. In today's factory automation, robots are tediously programmed for every single action. This makes robots feasible only in high-volume manufacturing, where they are programmed once to manufacture thousands of copies of the same product.
According to the International Federation of Robotics [1] more than 75% of all current industrial robots are used by large manufacturers performing mass production. The two main sectors are automotive, which use about 50% of all industrial robots, and mass-produced electronics (smart phones and tablets) which use about 25% of all industrial robots. SMEs, even though they contribute 47% of the turnover of UK's private sector [2], make little use of robotic technology. These manufacturers include metals, wood, plastics, and machinery sectors with tasks such as cutting, drilling, grinding, milling, and polishing of products made of metals, wood, plastics and others. The high-variety/low-volume nature of SME manufacturing makes it infeasible to install and re-program new robotic systems every time a product or task is changed. This makes flexible manufacturing, i.e. the ability to adapt to new products/tasks easily, a desired feature in automation.
Intelligent planning algorithms, by enabling robots to adapt to new tasks without the need for re-programming, has the potential to make a major contribution to flexible automation. By enabling flexible robotic automation, intelligent manipulation planning can contribute to improved productivity, reduced costs, and competitiveness for SMEs. We will organize activities within this project to make the planners we develop visible to its potential beneficiaries; manufacturers in the metals, wood, plastics, and machinery sectors. Specifically, I propose to:
* Create an Interactive Robot Demonstration, which will use our robot set-up to show the planner's capabilities to manufacturers in an interactive way;
* Organize an Industry Day, which will bring manufacturers to our lab to learn about our system and to give feedback;
* Create ROS-Industrial Software Packages, which will provide potential users with an easy way to try our algorithms on their systems.
The details of these activities are presented in "Pathways to Impact".
[1] http://www.ifr.org/industrial-robots/statistics/
[2] https://www.gov.uk/government/statistics/business-population-estimates-2015
Manufacturers which are small and medium-sized enterprises (SMEs) will benefit most from the intelligent planning technologies proposed here. In today's factory automation, robots are tediously programmed for every single action. This makes robots feasible only in high-volume manufacturing, where they are programmed once to manufacture thousands of copies of the same product.
According to the International Federation of Robotics [1] more than 75% of all current industrial robots are used by large manufacturers performing mass production. The two main sectors are automotive, which use about 50% of all industrial robots, and mass-produced electronics (smart phones and tablets) which use about 25% of all industrial robots. SMEs, even though they contribute 47% of the turnover of UK's private sector [2], make little use of robotic technology. These manufacturers include metals, wood, plastics, and machinery sectors with tasks such as cutting, drilling, grinding, milling, and polishing of products made of metals, wood, plastics and others. The high-variety/low-volume nature of SME manufacturing makes it infeasible to install and re-program new robotic systems every time a product or task is changed. This makes flexible manufacturing, i.e. the ability to adapt to new products/tasks easily, a desired feature in automation.
Intelligent planning algorithms, by enabling robots to adapt to new tasks without the need for re-programming, has the potential to make a major contribution to flexible automation. By enabling flexible robotic automation, intelligent manipulation planning can contribute to improved productivity, reduced costs, and competitiveness for SMEs. We will organize activities within this project to make the planners we develop visible to its potential beneficiaries; manufacturers in the metals, wood, plastics, and machinery sectors. Specifically, I propose to:
* Create an Interactive Robot Demonstration, which will use our robot set-up to show the planner's capabilities to manufacturers in an interactive way;
* Organize an Industry Day, which will bring manufacturers to our lab to learn about our system and to give feedback;
* Create ROS-Industrial Software Packages, which will provide potential users with an easy way to try our algorithms on their systems.
The details of these activities are presented in "Pathways to Impact".
[1] http://www.ifr.org/industrial-robots/statistics/
[2] https://www.gov.uk/government/statistics/business-population-estimates-2015
People |
ORCID iD |
Mehmet Dogar (Principal Investigator) |
Publications
Roa M
(2021)
Mobile Manipulation Hackathon: Moving into Real World Applications
in IEEE Robotics & Automation Magazine
Papallas R
(2020)
Online Replanning With Human-in-the-Loop for Non-Prehensile Manipulation in Clutter - A Trajectory Optimization Based Approach
in IEEE Robotics and Automation Letters
Papallas R
(2020)
Non-Prehensile Manipulation in Clutter with Human-In-The-Loop
Papallas R
(2020)
Non-Prehensile Manipulation in Clutter with Human-In-The-Loop
Papallas R
(2020)
Human-Guided Planner for Non-Prehensile Manipulation
Papallas R
(2019)
Non-Prehensile Manipulation in Clutter with Human-In-The-Loop
Figueredo L
(2021)
Human Comfortability: Integrating Ergonomics and Muscular-Informed Metrics for Manipulability Analysis During Human-Robot Collaboration
in IEEE Robotics and Automation Letters
Figueredo L
(2021)
Planning to Minimize the Human Muscular Effort during Forceful Human-Robot Collaboration
in ACM Transactions on Human-Robot Interaction
Chen L
(2020)
Manipulation planning under changing external forces
in Autonomous Robots
Chen L
(2018)
Manipulation Planning Under Changing External Forces
Chen L
(2018)
Manipulation Planning under Changing External Forces
Chen L
(2017)
Manipulation Planning under Changing External Forces
Agboh W
(2020)
Parareal with a learned coarse model for robotic manipulation
in Computing and Visualization in Science
Agboh W
(2022)
Robotics Research - The 19th International Symposium ISRR
Description | We have developed an algorithm for a robot to plan grasping an object such that the object can resist forceful operations applied on it, such as drilling and cutting. A video can be seen here: https://www.youtube.com/watch?v=IHti307yGFY We have also extended this algorithm such that it can be a human performing these forceful operations, and the algorithm considers the comfort of the human during the activity. |
Exploitation Route | With further improvement, such a system can be used in manufacturing/assembly environments, where a workpiece undergoes multiple operations, such as drilling, cutting, milling, polishing, etc. |
Sectors | Aerospace Defence and Marine Electronics Manufacturing including Industrial Biotechology |
URL | https://www.youtube.com/watch?v=IHti307yGFY |
Description | Findings of this project led to discussions with companies in the industry (companies including Amazon Robotics, Cavendish Nuclear, Advanced Supply Chain Group, Zebra Technologies, Bosch Research, Asda Logistics). These discussions led to new proposed projects on using robots in manufacturing and warehouses. One such project was funded, with support from Amazon Robotics and Advanced Supply Chain Group. This project is currently active, and the goal is to make the UK competitive in robotic automation of warehouse picking and packing. This is an area with increasing activity in many leading countries, and the outputs of this project help the UK to stay at the cutting edge of technology in automated warehouse picking and packing. We also have recently engaged in a meeting with Microsoft Research, where we presented findings related to this project, and the possibility of using robotic manipulation technologies in Data Centre Maintenance. Personnel from this project went on to be employed by leading Robotics companies (e.g., Tencent Robotics) to focus on projects related to robotic manipulation in manufacturing. |
First Year Of Impact | 2022 |
Sector | Electronics,Manufacturing, including Industrial Biotechology |
Impact Types | Economic |
Description | Robotic picking and packing with physical reasoning |
Amount | £1,196,800 (GBP) |
Funding ID | EP/V052659/1 |
Organisation | Engineering and Physical Sciences Research Council (EPSRC) |
Sector | Public |
Country | United Kingdom |
Start | 12/2021 |
End | 11/2026 |
Description | Multi-object manipulation (with UC Berkeley) |
Organisation | University of California, Berkeley |
Country | United States |
Sector | Academic/University |
PI Contribution | This is a collaboration between our group at University of Leeds and Prof. Ken Goldberg's group at UC Berkeley, USA, who are also partners of my EPSRC Fellowship. We have contributed to the development of methods for the manipulation and grasping of mutiple, rigid and deformable, objects by robots. We developed an initial prototype of the multi-object grasping system in Leeds. The system was further developed at UC Berkeley, together with a visitor from our team, and was recently generalized to deformable objects (e.g., garments/clothes). We also contributed with help to the writing of publications. |
Collaborator Contribution | The UC Berkeley team developed the core system as well as the algorithms methods, in collaboration with our team. They have performed most of the experimental work in their labs. They also led the writing of the publications. |
Impact | The collaboration resulted in four publications so far. |
Start Year | 2021 |
Description | Physics-based model-predictive control for object manipulation (with ETH Zurich) |
Organisation | ETH Zurich |
Country | Switzerland |
Sector | Academic/University |
PI Contribution | I visited Prof. Robert Katzschmann's group in ETH Zurich, who are also partners on my EPSRC Fellowship. During the visit, I learned about the tendon-driven, anthropomorphic robotic hand developed by Prof. Katzschmann's group, as well as the different learning/AI methods they are exploring to perform autonomous manipulation with this hand. I have contributed with inputs to the developments of such methods, particularly leading a method based on physics-based model-predictive control. This work is still ongoing. |
Collaborator Contribution | Prof. Katzschmann's group contributed by providing the robotic hand hardware, as well as all other necessary tools for the development. |
Impact | This collaboration does not have any outputs yet. It is ongoing work. |
Start Year | 2023 |
Description | Planning for human comfort during human-robot collaborative manipulation |
Organisation | University of Leeds |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | This was a collaboration with Dr Samit Chakrabarty from School of Biomedical Engineering, University of Leeds. We extended our robotic planners to include musculoskeletal definitions of human comfort, so that robots can choose actions that are comfortable for humans (e.g. where to hold an object a robot and a human are jointly manipulating). |
Collaborator Contribution | They helped us with definitions of human comfort and experiments to measure human muscle activity via EMG during human-robot collaboration. |
Impact | This collaboration led to a publication at the top-ranked journal in human-robot interaction (ACM Transactions in HRI), which then was invited for an oral presentation also at the top-ranked ACM/IEEE International Conference on Human-Robot Interaction, 2023, in Stockholm. We are also in the process of writing a follow-up joint grant proposal with Dr Chakrabarty. |
Start Year | 2018 |
Description | Use of tactile sensors in physics-based manipulation (with Univ. of Bristol) |
Organisation | University of Bristol |
Country | United Kingdom |
Sector | Academic/University |
PI Contribution | Integrating tactile sensors developed by Prof. Nathan Lepora's group at University of Bristol (who are partners on my EPSRC Fellowship) into our physics-based object perception algorithms and methods. |
Collaborator Contribution | Prof. Nathan Lepora's group contributed by teaching us how to manufacture one of their tactip sensors, including an on-site training of the PDRA, and also help with the software that accompanies this sensor. |
Impact | Ongoing collaboration. No outputs yet. |
Start Year | 2023 |
Description | 3rd UK Robot Manipulation Workshop |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Other audiences |
Results and Impact | We organized the 3rd UK Robot Manipulation Workshop in Leeds, with attendance from more than 100 researchers and industrial practitioners from all around the UK. The two-day event included talks by leading researchers in the field, and poster presentations from postgraduate researchers. The event website can be found here: https://www.robot-manipulation.uk/ |
Year(s) Of Engagement Activity | 2019 |
URL | https://www.robot-manipulation.uk/ |
Description | Industry visits |
Form Of Engagement Activity | A formal working group, expert panel or dialogue |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Industry/Business |
Results and Impact | We had companies, including Unilever, Rakusens Crackers, and others to see our robotic manipulation research to get informed about current technology on the potential of intelligent robotics in manufacturing. |
Year(s) Of Engagement Activity | 2018 |
Description | Open Days for Undergrad Applicants and Parents |
Form Of Engagement Activity | Participation in an open day or visit at my research institution |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Schools |
Results and Impact | We had multiple such Open Days / Applicant Days where we demonstrated our research. This has increased interest in our School and our Robotics activity. |
Year(s) Of Engagement Activity | 2018 |
Description | Visit by ASDA Logistics Services |
Form Of Engagement Activity | A formal working group, expert panel or dialogue |
Part Of Official Scheme? | No |
Geographic Reach | Regional |
Primary Audience | Industry/Business |
Results and Impact | A visit was performed by Asda Logistics Services to our labs to discuss possible applications of our object manipulation methods, in Asda warehouses. This was then followed by a visit from our researchers to Asda warehouses. |
Year(s) Of Engagement Activity | 2023 |