3D Vision Assisted Robotic Harvesting of Broccoli
Lead Research Organisation:
University of Lincoln
Department Name: School of Computer Science
Abstract
There is an urgent need to reduce the costs of production of field brassica crops, in particular broccoli. Labour costs are a significant proportion of overall production costs, high labour usage also drives complex management and potentially social issues. In this project we will test whether low-cost commercial 3D camera technology can be used to identify and select broccoli which are ready to harvest within commercial crops. This will provide a key underpinning step towards the development of a fully automatic and camera guided robotic harvesting system for broccoli. The commercial benefits are highly significant, the broccoli crop is one of the worlds largest vegetable crops, and almost all of it is manually harvested.
Technical Summary
There is an urgent need to reduce the costs of production of field brassica crops, in particular broccoli. Labour costs are a significant proportion of overall production costs, high labour usage also drives complex management and potentially social issues. In this project we will test whether low-cost commercial 3D camera technology can be used to identify and select broccoli which are ready to harvest within commercial crops. This will provide a key underpinning step towards the development of a fully automatic and camera guided robotic harvesting system for broccoli. The commercial benefits are highly significant, the broccoli crop is one of the worlds largest vegetable crops, and almost all of it is manually harvested.
Planned Impact
The project has the potential to derive a high level of economic and social benefit. The economic benefits will be exploited by the growers of broccoli within the UK. The social benefits will be in terms of public health, in that it will control the costs of broccoli selling price, enticing consumers to purchase a healthy food. It will reduce the need for manual labour to conduct a highly laborious, difficult and physically demanding job. These conditions result if increasing recruitment difficulties for growers, and absorb high levels of migrant workers. Given that the broccoli production industry is clustered in very few locations, the labour influx to relatively small settlements can place high pressure on local services.
The impact of this proposal in terms of the industry and social consequences are very high, but the project carries some risk as the main challenge has yet to be resolved. On this basis we believe that the project is highly appropriate for public funding from Innovate, and should provide a high level of return. There is no doubt that there are many challenges which the UK agriculture face in terms of developing improved vision analysis and robotic harvesting systems. This project will help develop capacity in a key industry area which is required to achieve the general government objective of the sustainable intensification of UK agriculture. A key positive step of this proposal is that is facilitates the transfer of the engineering, computing and robotics skills of the University of Lincoln into R Fountain and Son.
The impact of this proposal in terms of the industry and social consequences are very high, but the project carries some risk as the main challenge has yet to be resolved. On this basis we believe that the project is highly appropriate for public funding from Innovate, and should provide a high level of return. There is no doubt that there are many challenges which the UK agriculture face in terms of developing improved vision analysis and robotic harvesting systems. This project will help develop capacity in a key industry area which is required to achieve the general government objective of the sustainable intensification of UK agriculture. A key positive step of this proposal is that is facilitates the transfer of the engineering, computing and robotics skills of the University of Lincoln into R Fountain and Son.
Publications
Kusumam K
(2017)
3D-vision based detection, localization, and sizing of broccoli heads in the field
in Journal of Field Robotics
| Description | The project has demonstrated the feasibility of using low-cost RGB-D camera technology for 3D-vision-assisted robotic harvesting of broccoli, including the tasks of detecting mature broccoli heads in the field and providing their locations in 3D world coordinates. |
| Exploitation Route | Following successful demonstration of feasibility, we are looking for opportunities to develop a full robotic harvesting machine based on the developed technology. |
| Sectors | Agriculture Food and Drink |
| Description | The project has attracted considerable media and press interest, including being praised as 'world leading' by UK Farming Minister George Eustice. This in turn has influenced the competition, and we are aware of several current projects attempting to utilise 3D imaging for broccoli harvesting. It has also helped to influence public debate on the potential for robotic automation of agriculture, as reported in many outlets in the popular press. |
| First Year Of Impact | 2015 |
| Sector | Agriculture, Food and Drink |
| Impact Types | Societal Economic |
| Description | AHDB tender CP 153a Development of a selective broccoli harvester |
| Amount | £257,656 (GBP) |
| Organisation | Agricultural and Horticulture Development Board |
| Sector | Charity/Non Profit |
| Country | United Kingdom |
| Start | 05/2016 |
| End | 06/2017 |
| Title | Broccoli dataset (UK and Spain) |
| Description | The four data collection sessions were conducted at three different sites in Lincolnshire, UK and one in Murcia, Spain (see Fig. 1). The sessions were conducted at the beginning and towards the end of harvesting season in UK and at the end of the harvest in Spain. The variety of broccoli plants grown in UK is called Iron Man whilst the variety grown in Spain is called Titanium.The weather during UK data capture included a mixture of different conditions including sunny, overcast and raining with broccoli varying in maturity levels from small to larger to already harvested, while the conditions for data capture in Spain included strong sunlight and mature plants at the very end of the harvesting season. The tractor was driven through the broccoli field at a slow walking speed with two rows of broccoli plants being imaged by the RGB-D sensor. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2016 |
| Provided To Others? | Yes |
| Impact | The publication of the dataset led to a journal publication in Journal of Field Robotics and has been used already by other institutions (e.g. Politecnico di Milano). |
| URL | https://lcas.lincoln.ac.uk/owncloud/shared/agritech-datasets/broccoli/broccoli_datasets.html |
| Description | Industrial collaborator on the grant |
| Organisation | R. Fountain and Son Ltd. |
| Country | United Kingdom |
| Sector | Private |
| PI Contribution | We worked together with Fountains to deliver a feasibility study on 3D Vision-Assisted Robotic Harvesting of Broccoli |
| Collaborator Contribution | Mechanical and electrical design of the image capture set-up for the project, including enclosure and lighting set-up, and producing the concept design for the broccoli cutting equipment. Facilitating and assisting data collection in the field, including trials in Lincolnshire, UK, and Murcia, Spain. |
| Impact | A prototype image capture set-up was developed and refined during the project. |
| Start Year | 2015 |
| Title | Algorithms for broccoli head detection and localisation |
| Description | Research software has been developed using data from low-cost RGB-D camera for detecting mature broccoli heads in the field and providing their locations in 3D world coordinates. The system could potentially be used to differentiate mature broccoli heads and applied to an autonomous harvester for the selective harvesting of broccoli crops. |
| Type Of Technology | Software |
| Year Produced | 2016 |
| Impact | The developments are currently under evaluation, which includes data collection in both UK and Spain, which have complementary growing seasons. Papers have been submitted to relevant conferences and work continues. |
| Description | Agri-Tech East Pollinator: Robo-Cropping - The Potential for Precision Robotics in Arable Agriculture |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Industry/Business |
| Results and Impact | This Pollinator will explore the technologies that are in development or already in use to increase the efficiency and accuracy of crop production, provide insights into the latest innovations that are ready to hit the fields, and offer technology developers industry insights into what farmers really need in a robot. |
| Year(s) Of Engagement Activity | 2016 |
| URL | https://www.cambridgenetwork.co.uk/events/agri-tech-east-pollinator-robo-cropping-the-potential/ |
| Description | Farmers Weekly Arable Horizons event - Interactive talks exploring tomorrow's farming technology |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Industry/Business |
| Results and Impact | Professor Simon Pearson and Professor Tom Duckett from the Lincoln Institute for Agri-food Technology (LIAT) at the University of Lincoln, UK, led the Robotics, Automation and Associated Technologies event taking place at the National Space Centre in Leicester on Wednesday 26th April 2017. The event was free to attend and delegates could book their place online. Together they detailed the pioneering research taking place at the University of Lincoln and explained to industry guests how robots and other novel technologies can make UK farming more efficient than ever, while reducing the cost of labour and use of pesticides. The event was also broadcast live on the internet and on youtube. |
| Year(s) Of Engagement Activity | 2017 |
| URL | https://www.youtube.com/watch?v=sp1JzDmOBFo&t=1998s |
| Description | Hosting visit by representatives of NatWest bank including presentation and discussion of research in robotic agriculture at the University of Lincoln, at the Isaac Newton Building, Brayford Pool, University of, Lincoln LN6 7TS on Thursday 23rd January 2020. |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Other audiences |
| Results and Impact | Hosting visit by representatives of NatWest bank including presentation and discussion of research in robotic agriculture at the University of Lincoln, at the Isaac Newton Building, Brayford Pool, University of, Lincoln LN6 7TS on Thursday 23rd January 2020. This sparked many questions and discussions, and a follow-up visit with national executives of the bank was subsequently requested by the visitors. |
| Year(s) Of Engagement Activity | 2020 |
| Description | Introductory talk: \Robotics Research at the University of Lincoln", Isaac Newton Building, University of Lincoln, 12th February 2019, for the Lincolnshire Robotics Forum, to representatives from local industry with interests in robotics, followed by discussions and lab tour, sponsored by NatWest Bank, Business Lincolnshire and ERDF |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Industry/Business |
| Results and Impact | Introductory talk: "Robotics Research at the University of Lincoln", Isaac Newton Building, University of Lincoln, 12th February 2019, for the Lincolnshire Robotics Forum, to representatives from local industry with interests in robotics, followed by discussions and lab tour, sponsored by NatWest Bank, Business Lincolnshire and ERDF. Sparked questions and discussions afterwards with potential industry partners. |
| Year(s) Of Engagement Activity | 2019 |
| Description | Invited talk at IROS 2016 |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | An invited talk about agricultural robotics at the workshop on "State Estimation and Terrain Perception for All Terrain Mobile Robots" held during the IROS 2016 conference |
| Year(s) Of Engagement Activity | 2016 |
| Description | Keynote talk "The Future of Robotic Agriculture", Nottingham Trent University, 9th January 2019, as part of the Smart Industry Workshop: Recent Advances in Industrial Digitalisation, Robotics and Automation, 9-11 January 2019. |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Postgraduate students |
| Results and Impact | Keynote talk "The Future of Robotic Agriculture", Nottingham Trent University, 9th January 2019, as part of the Smart Industry Workshop: Recent Advances in Industrial Digitalisation, Robotics and Automation, 9-11 January 2019. Sparked questions and discussions afterwards, including invitations to discuss joint research bids for future follow-on projects. |
| Year(s) Of Engagement Activity | 2019 |
| URL | http://smartindustry4.uk/ |
| Description | Keynote: "The Future of Robotic Agriculture", Loughborough University, 24th January 2019, UKRAS19: Embedded Intelligence, annual conference of the EPSRC UK-RAS Network. |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Postgraduate students |
| Results and Impact | Keynote: "The Future of Robotic Agriculture", Loughborough University, 24th January 2019, UKRAS19: Embedded Intelligence, annual conference of the EPSRC UK-RAS Network. Sparked questions and discussions afterwards, including a lot of interest in agri-robotics research at the University of Lincoln. |
| Year(s) Of Engagement Activity | 2019 |
| URL | https://www.ukras.org/news-and-events/uk-ras/ukras19-programme/ |
| Description | Presentation and discussion on the future of robotic agriculture at the Edenham & District Farmers' Discussion Group on Wednesday 9th October 2019 |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Industry/Business |
| Results and Impact | Around 40 people from the regional farming community attended this event organised by the Edenham & District Farmers' Discussion Group at the Toft House Hotel, Toft, Bourne PE10 0JT on the evening Wednesday 9th October 2019, which sparked questions and discussion afterwards. The event serves as a fora for discussing topical issues within the regional farming community. I also joined the organisers for dinner before the presentation and discussion with the wider Group. |
| Year(s) Of Engagement Activity | 2019 |
| Description | Presentation and discussion on the use of robotics and autonomous systems on the farm at the Lincoln Institute for Agri-Food Technology Alumni Event, University of Lincoln, Riseholme Park, Lincoln LN2 2LG on Thursday 6th February 2020. |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | Presentation and discussion on the use of robotics and autonomous systems on the farm at the Lincoln Institute for Agri-Food Technology Alumni Event ,University of Lincoln, Riseholme Park, Lincoln LN2 2LG on Thursday 6th February 2020, which sparked questions and discussions afterwards. |
| Year(s) Of Engagement Activity | 2020 |
| Description | Presentation of 3D Vision-Assisted Robotic Harvesting of Broccoli |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Industry/Business |
| Results and Impact | A presentation and discussion with the Institute of Agricultural Management at the National Centre for Food Manufacturing, Holbeach, Lincolnshire on 7th November 2016. |
| Year(s) Of Engagement Activity | 2016 |
| URL | https://www.iagrm.org.uk/ |
| Description | Public presentation and discussion on the future of robotic agriculture at the Pessimist Wine and Gin Bar, Mint Ln, Lincoln LN1 1UD on Tuesday 21st May 2019 as part of the Pint of Science Festival, a nation-wide public engagement activity involving UK universities. |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Public/other audiences |
| Results and Impact | Public presentation and discussion on the future of robotic agriculture at the Pessimist Wine and Gin Bar, Mint Ln, Lincoln LN1 1UD on Tuesday 21st May 2019 as part of the Pint of Science Festival, a nation-wide public engagement activity involving UK universities, which sparked questions and discussion afterwards. |
| Year(s) Of Engagement Activity | 2019 |
| Description | Research Seminar on "The Future of Robotic Agriculture" at University of Manchester |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Postgraduate students |
| Results and Impact | Research Seminar: "The Future of Robotic Agriculture", School of Electrical and Electronic Engineering, University of Manchester, 5th December 2018, invited by the University of Manchester Robotics Group. This was attended by N8 PhD students in Agri-Food as well as interested academics and researchers. |
| Year(s) Of Engagement Activity | 2018 |
| Description | Stakeholder meeting on 10 Nov 2015 at National Centre for Food Manufacturing. |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Industry/Business |
| Results and Impact | The first stake-holder meeting in the project was co-located with the "Showcasing Developments in Agri-Food Technology" event on 10th November 2015, National Centre for Food Manufacturing, Holbeach, Lincolnshire, as part of Agri-Tech Week 2015 (9-15 November 2015) organised by Agri-Tech East. The event was attended by 45 persons including Andy Blair of TH Clements Ltd., a major UK grower of broccoli and representative of the project advisory board. |
| Year(s) Of Engagement Activity | 2015 |
| URL | http://www.greaterlincolnshirelep.co.uk/events/view/showcasing-developments-in-agri-food-technology |