16AGRITECHCAT5: 3D Vision-based Crop-Weed Discrimination for Automated Weeding Operations
Lead Research Organisation:
University of Lincoln
Department Name: School of Computer Science
Abstract
This project will investigate technical foundations for the next generation of robotic weeding machinery enabling selective and accurate treatment of individual weeds. The investigated technology is a novel combination of low-cost 3d sensing and learning software together with a suitable weed destruction technique. The proposed developments will lead to more efficient cultural weeding equipment resulting in better management of weeds and reduced input use bringing several benefits to good producers, sellers and society.
Technical Summary
This project will investigate technical foundations for the next generation of robotic weeding machinery enabling selective and accurate treatment of individual weeds. The investigated technology is a novel combination of low-cost 3d sensing and learning software together with a suitable weed destruction technique. The proposed developments will lead to more efficient cultural weeding equipment resulting in better management of weeds and reduced input use bringing several benefits to good producers, sellers and society.
Planned Impact
The project has the potential to derive a high level of economic and social benefit. The proposed technological solution for robotic cultural weeding machinery will result in reduced input use and increased efficiency in primary crop production and indirectly also in sustained food quality and safety. The potential products will have positive impact on the entire UK Agri-Tech sector and its one representative in particular, Garford Farm Machinery. The economic benefits will be also felt by food producers, sellers and consumers.
The impact of this proposal in terms of the industry and social consequences are very high, but the project carries some risk as the main challenge has yet to be resolved. On this basis we believe that the project is highly appropriate for public funding from Innovate, and should provide a high level of return. There is no doubt that there are many challenges which the UK agriculture face in terms of developing improved vision analysis and robotic weeding systems. This project will help develop further capacity in a key industry area which is required to achieve the general government objective of the sustainable intensification of UK agriculture. A key positive step of this proposal is that is facilitates the transfer of the robotic and computing skills of the University of Lincoln into Garford Farm Machinery Ltd.
The impact of this proposal in terms of the industry and social consequences are very high, but the project carries some risk as the main challenge has yet to be resolved. On this basis we believe that the project is highly appropriate for public funding from Innovate, and should provide a high level of return. There is no doubt that there are many challenges which the UK agriculture face in terms of developing improved vision analysis and robotic weeding systems. This project will help develop further capacity in a key industry area which is required to achieve the general government objective of the sustainable intensification of UK agriculture. A key positive step of this proposal is that is facilitates the transfer of the robotic and computing skills of the University of Lincoln into Garford Farm Machinery Ltd.
Publications
Bosilj P
(2018)
Connected attribute morphology for unified vegetation segmentation and classification in precision agriculture.
in Computers in industry
Bosilj P
(2018)
Analysis of Morphology-Based Features for Classification of Crop and Weeds in Precision Agriculture
in IEEE Robotics and Automation Letters
Bosilj P
(2019)
Transfer learning between crop types for semantic segmentation of crops versus weeds in precision agriculture
in Journal of Field Robotics
| Description | The main aim of the project was to investigate the technical foundations for the next generation of robotic weeding machinery, enabling selective and accurate treatment of specific weeds. The technical objectives of the project included detection of plants using a low-cost 3D camera vision system, discrimination of target crop plants from weeds at different growth stages, providing an intuitive system training interface for rapid deployment, development of a proof-of-concept weed destruction technique and finally integration and evaluation of the developed technology in automated weeding products. The project successfully demonstrated a proof-of-concept computer vision pipeline that enables successful discrimination between crop plants (i.e. carrots and onions) and weeds. During the project, the team concluded that current 3D vision sensing technology provides reliable location estimation of large plants but is unsuitable to cope with very small plants and therefore the project investigated an alternative sensing approach based on multi-spectral sensing. The new sensing principle proved to be reliable for plants in the early stages of growth before the full canopy closure. Machine learning-enabled robustness to plant size and variation and quick retraining for specific target crop. A basic user training interface was created but further work in that area is required to streamline the operation with an inexperienced operator. The project investigated multiple weeding techniques including two types of mechanical hoes, electric charge, high-pressure water jet and plasma cutter. The most promising mechanical hoe option was then selected for further investigation. The project resulted in a demonstrable mechanical hoe with very good soil engagement parameters and responsiveness to control input. A prototype demonstrator combining a simple vision system and the new hoe design was built and evaluated providing the basis for further investigation and development. The project resulted in a large database of images taken from various fields of carrots and onions at the different growth stage. This will be used for further developments, algorithm training, etc. The new multi-spectral sensing principle has the potential for straightforward inclusion of 3D information by adding one more machine vision camera. This would result in a relatively inexpensive sensing apparatus combining the strengths of multi-spectral and 3D vision. The issue of scalability of the hoeing apparatus was looked at and the potential for the development of small weeding robots was considered for further investigation. This would open completely new markets and services and collaboration with robotic companies. Freedom to operate was fully assessed with the conclusion that it is not restricted by current developments, although rapid progress is being made by competitors and therefore further R&D work is essential. |
| Exploitation Route | To translate the R&D effort undertaken in the feasibility study into the existing weeding products additional engineering efforts are required. This includes: - identification of a suitable hardware platform; - translation and deployment of the current software prototype into the industrial demonstrator; - full integration of the vision system with the control component of the mechanical hoe; - validation of the entire vision-based control loop in field conditions. |
| Sectors | Agriculture Food and Drink |
| Description | The project has attracted significant media attention (40+ releases, including BBC Look North, Countryfile) and robotic weeding has been promoted to general public in 30+ industry fairs and exhibitions (Cereals, Oco Field Day, Norfolk Day, Agritechnica, Croptec, Stakeholder meeting at BOG Conference). |
| First Year Of Impact | 2017 |
| Sector | Agriculture, Food and Drink,Digital/Communication/Information Technologies (including Software) |
| Impact Types | Societal Economic |
| Description | BBSRC Seeding Catalyst |
| Amount | £26,000 (GBP) |
| Organisation | Biotechnology and Biological Sciences Research Council (BBSRC) |
| Sector | Public |
| Country | United Kingdom |
| Start | 11/2017 |
| End | 02/2018 |
| Description | Development and field testing of the next generation of vision-guided weeding systems |
| Amount | £697,058 (GBP) |
| Funding ID | 26507 |
| Organisation | Innovate UK |
| Sector | Public |
| Country | United Kingdom |
| Start | 04/2019 |
| End | 04/2021 |
| Description | Impact Acceleration Fund |
| Amount | £7,000 (GBP) |
| Organisation | University of Lincoln |
| Sector | Academic/University |
| Country | United Kingdom |
| Start | 03/2018 |
| End | 05/2018 |
| Title | Onion and carrot image dataset |
| Description | A set of annotated images with crops and weeds. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2018 |
| Provided To Others? | No |
| Impact | Not available yet. |
| Description | New collaboration with CBS Ltd. on autonomous weeding robots |
| Organisation | CBS Corporation |
| Country | United States |
| Sector | Private |
| PI Contribution | Work on software components for the autonomous mobile robot Thorvald. An initial design idea integrating Thorvald with weeding implements manufactured by GFM. |
| Collaborator Contribution | GFM: implement development. CBS: autonomous platform Thorvald. |
| Impact | A successful followup grant: "Development and field testing of the next generation of vision-guided weeding systems", funded by Innovate UK (#26507) in 2019. |
| Start Year | 2017 |
| Title | Algorithms for weed/crop discrimination |
| Description | Research software has been developed to perform weed/crop discrimination from multi-spectral image data (RGB and NIR spectra). The software was prototyped in Python and forms the basis for the next generation of software components which will be developed in the follow up project on autonomous weeding robots. |
| Type Of Technology | Software |
| Year Produced | 2018 |
| Impact | Papers have been submitted to relevant journals and conferences and work continues as part of the newly awarded project on autonomous weeding robots. |
| Description | Collaboration Nation |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Presentation of successful IUK/BBSRC funded projects at the Collaboration Nation workshop. March 2018 |
| Year(s) Of Engagement Activity | 2018 |
| Description | Farmers Weekly Arable Horizons event - Interactive talks exploring tomorrow's farming technology |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Industry/Business |
| Results and Impact | Professor Simon Pearson and Professor Tom Duckett from the Lincoln Institute for Agri-food Technology (LIAT) at the University of Lincoln, UK, led the Robotics, Automation and Associated Technologies event taking place at the National Space Centre in Leicester on Wednesday 26th April 2017. The event was free to attend and delegates could book their place online. Together they detailed the pioneering research taking place at the University of Lincoln and explained to industry guests how robots and other novel technologies can make UK farming more efficient than ever, while reducing the cost of labour and use of pesticides. The event was also broadcast live on the internet and on youtube. |
| Year(s) Of Engagement Activity | 2017 |
| URL | https://www.youtube.com/watch?v=sp1JzDmOBFo&t=1998s |
| Description | Future Fest at the University of Lincoln |
| Form Of Engagement Activity | Participation in an open day or visit at my research institution |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Public/other audiences |
| Results and Impact | Dissemination to general public during Future Fest at the University of Lincoln, Oct, 2016 |
| Year(s) Of Engagement Activity | 2016 |
| Description | Hosting visit by representatives of NatWest bank including presentation and discussion of research in robotic agriculture at the University of Lincoln, at the Isaac Newton Building, Brayford Pool, University of, Lincoln LN6 7TS on Thursday 23rd January 2020. |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Other audiences |
| Results and Impact | Hosting visit by representatives of NatWest bank including presentation and discussion of research in robotic agriculture at the University of Lincoln, at the Isaac Newton Building, Brayford Pool, University of, Lincoln LN6 7TS on Thursday 23rd January 2020. This sparked many questions and discussions, and a follow-up visit with national executives of the bank was subsequently requested by the visitors. |
| Year(s) Of Engagement Activity | 2020 |
| Description | Introductory talk: \Robotics Research at the University of Lincoln", Isaac Newton Building, University of Lincoln, 12th February 2019, for the Lincolnshire Robotics Forum, to representatives from local industry with interests in robotics, followed by discussions and lab tour, sponsored by NatWest Bank, Business Lincolnshire and ERDF |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Industry/Business |
| Results and Impact | Introductory talk: "Robotics Research at the University of Lincoln", Isaac Newton Building, University of Lincoln, 12th February 2019, for the Lincolnshire Robotics Forum, to representatives from local industry with interests in robotics, followed by discussions and lab tour, sponsored by NatWest Bank, Business Lincolnshire and ERDF. Sparked questions and discussions afterwards with potential industry partners. |
| Year(s) Of Engagement Activity | 2019 |
| Description | Invited talk at IROS 2016 |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | An invited talk about agricultural robotics at the workshop on "State Estimation and Terrain Perception for All Terrain Mobile Robots" held during the IROS 2016 conference |
| Year(s) Of Engagement Activity | 2016 |
| Description | Keynote talk "The Future of Robotic Agriculture", Nottingham Trent University, 9th January 2019, as part of the Smart Industry Workshop: Recent Advances in Industrial Digitalisation, Robotics and Automation, 9-11 January 2019. |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Postgraduate students |
| Results and Impact | Keynote talk "The Future of Robotic Agriculture", Nottingham Trent University, 9th January 2019, as part of the Smart Industry Workshop: Recent Advances in Industrial Digitalisation, Robotics and Automation, 9-11 January 2019. Sparked questions and discussions afterwards, including invitations to discuss joint research bids for future follow-on projects. |
| Year(s) Of Engagement Activity | 2019 |
| URL | http://smartindustry4.uk/ |
| Description | Keynote: "The Future of Robotic Agriculture", Loughborough University, 24th January 2019, UKRAS19: Embedded Intelligence, annual conference of the EPSRC UK-RAS Network. |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Postgraduate students |
| Results and Impact | Keynote: "The Future of Robotic Agriculture", Loughborough University, 24th January 2019, UKRAS19: Embedded Intelligence, annual conference of the EPSRC UK-RAS Network. Sparked questions and discussions afterwards, including a lot of interest in agri-robotics research at the University of Lincoln. |
| Year(s) Of Engagement Activity | 2019 |
| URL | https://www.ukras.org/news-and-events/uk-ras/ukras19-programme/ |
| Description | Onion & Carrot Conference 2017 |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | Petra Bosilj gave a talk presenting the findings from the project. |
| Year(s) Of Engagement Activity | 2017 |
| Description | Presentation and discussion on the future of robotic agriculture at the Edenham & District Farmers' Discussion Group on Wednesday 9th October 2019 |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Industry/Business |
| Results and Impact | Around 40 people from the regional farming community attended this event organised by the Edenham & District Farmers' Discussion Group at the Toft House Hotel, Toft, Bourne PE10 0JT on the evening Wednesday 9th October 2019, which sparked questions and discussion afterwards. The event serves as a fora for discussing topical issues within the regional farming community. I also joined the organisers for dinner before the presentation and discussion with the wider Group. |
| Year(s) Of Engagement Activity | 2019 |
| Description | Presentation and discussion on the use of robotics and autonomous systems on the farm at the Lincoln Institute for Agri-Food Technology Alumni Event, University of Lincoln, Riseholme Park, Lincoln LN2 2LG on Thursday 6th February 2020. |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | Presentation and discussion on the use of robotics and autonomous systems on the farm at the Lincoln Institute for Agri-Food Technology Alumni Event ,University of Lincoln, Riseholme Park, Lincoln LN2 2LG on Thursday 6th February 2020, which sparked questions and discussions afterwards. |
| Year(s) Of Engagement Activity | 2020 |
| Description | Public presentation and discussion on the future of robotic agriculture at the Pessimist Wine and Gin Bar, Mint Ln, Lincoln LN1 1UD on Tuesday 21st May 2019 as part of the Pint of Science Festival, a nation-wide public engagement activity involving UK universities. |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Public/other audiences |
| Results and Impact | Public presentation and discussion on the future of robotic agriculture at the Pessimist Wine and Gin Bar, Mint Ln, Lincoln LN1 1UD on Tuesday 21st May 2019 as part of the Pint of Science Festival, a nation-wide public engagement activity involving UK universities, which sparked questions and discussion afterwards. |
| Year(s) Of Engagement Activity | 2019 |
| Description | REAP conference in Cambridge |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Regional |
| Primary Audience | Professional Practitioners |
| Results and Impact | Dissemination to general public at the REAP conference in Cambridge, Nov, 2016. |
| Year(s) Of Engagement Activity | 2016 |
| Description | Research Seminar on "The Future of Robotic Agriculture" at University of Manchester |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Postgraduate students |
| Results and Impact | Research Seminar: "The Future of Robotic Agriculture", School of Electrical and Electronic Engineering, University of Manchester, 5th December 2018, invited by the University of Manchester Robotics Group. This was attended by N8 PhD students in Agri-Food as well as interested academics and researchers. |
| Year(s) Of Engagement Activity | 2018 |