3D Vision Assisted Robotic Harvesting of Broccoli

Lead Research Organisation: University of Lincoln
Department Name: School of Computer Science

Abstract

There is an urgent need to reduce the costs of production of field brassica crops, in particular broccoli. Labour costs are a significant proportion of overall production costs, high labour usage also drives complex management and potentially social issues. In this project we will test whether low-cost commercial 3D camera technology can be used to identify and select broccoli which are ready to harvest within commercial crops. This will provide a key underpinning step towards the development of a fully automatic and camera guided robotic harvesting system for broccoli. The commercial benefits are highly significant, the broccoli crop is one of the worlds largest vegetable crops, and almost all of it is manually harvested.

Technical Summary

There is an urgent need to reduce the costs of production of field brassica crops, in particular broccoli. Labour costs are a significant proportion of overall production costs, high labour usage also drives complex management and potentially social issues. In this project we will test whether low-cost commercial 3D camera technology can be used to identify and select broccoli which are ready to harvest within commercial crops. This will provide a key underpinning step towards the development of a fully automatic and camera guided robotic harvesting system for broccoli. The commercial benefits are highly significant, the broccoli crop is one of the worlds largest vegetable crops, and almost all of it is manually harvested.

Planned Impact

The project has the potential to derive a high level of economic and social benefit. The economic benefits will be exploited by the growers of broccoli within the UK. The social benefits will be in terms of public health, in that it will control the costs of broccoli selling price, enticing consumers to purchase a healthy food. It will reduce the need for manual labour to conduct a highly laborious, difficult and physically demanding job. These conditions result if increasing recruitment difficulties for growers, and absorb high levels of migrant workers. Given that the broccoli production industry is clustered in very few locations, the labour influx to relatively small settlements can place high pressure on local services.

The impact of this proposal in terms of the industry and social consequences are very high, but the project carries some risk as the main challenge has yet to be resolved. On this basis we believe that the project is highly appropriate for public funding from Innovate, and should provide a high level of return. There is no doubt that there are many challenges which the UK agriculture face in terms of developing improved vision analysis and robotic harvesting systems. This project will help develop capacity in a key industry area which is required to achieve the general government objective of the sustainable intensification of UK agriculture. A key positive step of this proposal is that is facilitates the transfer of the engineering, computing and robotics skills of the University of Lincoln into R Fountain and Son.

Publications

10 25 50
 
Description The project has demonstrated the feasibility of using low-cost RGB-D camera technology for 3D-vision-assisted robotic harvesting of broccoli, including the tasks of detecting mature broccoli heads in the field and providing their locations in 3D world coordinates.
Exploitation Route Following successful demonstration of feasibility, we are looking for opportunities to develop a full robotic harvesting machine based on the developed technology.
Sectors Agriculture, Food and Drink

 
Description The project has attracted considerable media and press interest, including being praised as 'world leading' by UK Farming Minister George Eustice. This in turn has influenced the competition, and we are aware of several current projects attempting to utilise 3D imaging for broccoli harvesting. It has also helped to influence public debate on the potential for robotic automation of agriculture, as reported in many outlets in the popular press.
First Year Of Impact 2015
Sector Agriculture, Food and Drink
Impact Types Societal,Economic

 
Description AHDB tender CP 153a Development of a selective broccoli harvester
Amount £70,000 (GBP)
Organisation Agricultural and Horticulture Development Board 
Sector Public
Country United Kingdom of Great Britain & Northern Ireland (UK)
Start 06/2016 
End 06/2017
 
Title Broccoli dataset (UK and Spain) 
Description The four data collection sessions were conducted at three different sites in Lincolnshire, UK and one in Murcia, Spain (see Fig. 1). The sessions were conducted at the beginning and towards the end of harvesting season in UK and at the end of the harvest in Spain. The variety of broccoli plants grown in UK is called Iron Man whilst the variety grown in Spain is called Titanium.The weather during UK data capture included a mixture of different conditions including sunny, overcast and raining with broccoli varying in maturity levels from small to larger to already harvested, while the conditions for data capture in Spain included strong sunlight and mature plants at the very end of the harvesting season. The tractor was driven through the broccoli field at a slow walking speed with two rows of broccoli plants being imaged by the RGB-D sensor. 
Type Of Material Database/Collection of data 
Year Produced 2016 
Provided To Others? Yes  
Impact The publication of the dataset led to a journal publication in Journal of Field Robotics and has been used already by other institutions (e.g. Politecnico di Milano). 
URL https://lcas.lincoln.ac.uk/owncloud/shared/agritech-datasets/broccoli/broccoli_datasets.html
 
Description Industrial collaborator on the grant 
Organisation R. Fountain and Son Ltd.
Country United Kingdom of Great Britain & Northern Ireland (UK) 
Sector Private 
PI Contribution We worked together with Fountains to deliver a feasibility study on 3D Vision-Assisted Robotic Harvesting of Broccoli
Collaborator Contribution Mechanical and electrical design of the image capture set-up for the project, including enclosure and lighting set-up, and producing the concept design for the broccoli cutting equipment. Facilitating and assisting data collection in the field, including trials in Lincolnshire, UK, and Murcia, Spain.
Impact A prototype image capture set-up was developed and refined during the project.
Start Year 2015
 
Title Algorithms for broccoli head detection and localisation 
Description Research software has been developed using data from low-cost RGB-D camera for detecting mature broccoli heads in the field and providing their locations in 3D world coordinates. The system could potentially be used to differentiate mature broccoli heads and applied to an autonomous harvester for the selective harvesting of broccoli crops. 
Type Of Technology Software 
Year Produced 2016 
Impact The developments are currently under evaluation, which includes data collection in both UK and Spain, which have complementary growing seasons. Papers have been submitted to relevant conferences and work continues. 
 
Description Agri-Tech East Pollinator: Robo-Cropping - The Potential for Precision Robotics in Arable Agriculture 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Industry/Business
Results and Impact This Pollinator will explore the technologies that are in development or already in use to increase the efficiency and accuracy of crop production, provide insights into the latest innovations that are ready to hit the fields, and offer technology developers industry insights into what farmers really need in a robot.
Year(s) Of Engagement Activity 2016
URL https://www.cambridgenetwork.co.uk/events/agri-tech-east-pollinator-robo-cropping-the-potential/
 
Description Farmers Weekly Arable Horizons event - Interactive talks exploring tomorrow's farming technology 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Industry/Business
Results and Impact Professor Simon Pearson and Professor Tom Duckett from the Lincoln Institute for Agri-food Technology (LIAT) at the University of Lincoln, UK, led the Robotics, Automation and Associated Technologies event taking place at the National Space Centre in Leicester on Wednesday 26th April 2017. The event was free to attend and delegates could book their place online. Together they detailed the pioneering research taking place at the University of Lincoln and explained to industry guests how robots and other novel technologies can make UK farming more efficient than ever, while reducing the cost of labour and use of pesticides. The event was also broadcast live on the internet and on youtube.
Year(s) Of Engagement Activity 2017
URL https://www.youtube.com/watch?v=sp1JzDmOBFo&t=1998s
 
Description Invited talk at IROS 2016 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact An invited talk about agricultural robotics at the workshop on "State Estimation and Terrain Perception for All Terrain Mobile Robots" held during the IROS 2016 conference
Year(s) Of Engagement Activity 2016
 
Description Presentation of 3D Vision-Assisted Robotic Harvesting of Broccoli 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Industry/Business
Results and Impact A presentation and discussion with the Institute of Agricultural Management at the National Centre for Food Manufacturing, Holbeach, Lincolnshire on 7th November 2016.
Year(s) Of Engagement Activity 2016
URL https://www.iagrm.org.uk/
 
Description Stakeholder meeting on 10 Nov 2015 at National Centre for Food Manufacturing. 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Industry/Business
Results and Impact The first stake-holder meeting in the project was co-located with the "Showcasing Developments in Agri-Food Technology" event on 10th November 2015, National Centre for Food Manufacturing, Holbeach, Lincolnshire, as part of Agri-Tech Week 2015 (9-15 November 2015) organised by Agri-Tech East. The event was attended by 45 persons including Andy Blair of TH Clements Ltd., a major UK grower of broccoli and representative of the project advisory board.
Year(s) Of Engagement Activity 2015
URL http://www.greaterlincolnshirelep.co.uk/events/view/showcasing-developments-in-agri-food-technology