Plant Phenotyping as a multi task machine learning problem

Lead Research Organisation: University of Edinburgh
Department Name: Sch of Engineering

Abstract

Agricultural crop production is at risk for not keeping up with the growing demand given by world population growth and waste. To meet this expanding demand, plant breeders are focusing on finding more efficient plants which need to be grown in increasingly unfavourable environments as a consequence of climate change (Awlia et al., 2016). Plant phenotyping is a crucial step in this process as genetic variations need to be evaluated in terms of their phenotypic changes.
Machine learning is a powerful tool when trying to solve phenotyping problems. It allows scientists to utilize larger and more detailed datasets to discover patterns and drive discovery by measuring a combination of properties instead of analysing each one individually. Applications for plant phenotyping based on machine learning techniques have been made to segment plants (Minervini et al., 2014) as well as count leaves (Giuffrida, 2015).
My project will focus on implementing novel machine learning approaches and algorithms to solve current plant phenotyping problems. I plan to use multi-task deep learning and instance segmentation to detect and quantify different phenotypes such as leaf and plant growth as well as development. The objective is to build a robust system of object detection and tracking that could be used to phenotype multiple model plants such as Arabidopsis thaliana or Nicotiana tabacum.
Although plant phenotyping is an application in this project, the developed code and theory would be applicable to multiple other computer vision problems such as improved object detection and recognition for robotics or autonomous cars as they also must solve often complex and heavily occluded images.

Studentship Projects

Project Reference Relationship Related To Start End Student Name
EP/N509644/1 01/10/2016 30/09/2021
1785349 Studentship EP/N509644/1 01/10/2016 31/08/2020 Andrei Dobrescu
 
Description Image processing and computer vision have been an important field of computer science for decades. Importantly in the last few years, there has been an exponential growth of image and video data being generated and efficiently analysed using new methods of machine learning. This surge has driven industries and researchers in fields like agriculture and plant science and use image data to automate processes to save time and resources. One specific area of interest is plant phenotyping, which has been identified as one of the bottlenecks of agricultural research and is critical for the development of improved crops, necessary for ensuring global food security. My contributions have come from developing deep learning models applied to plant phenotyping problems. One of the main requirements surrounding deep learning approaches for computer vision is the need for large annotated datasets which are costly and time consuming to produce. My research focused as well to take advantage of weakly annotated or even unlabelled data to ease the burden of entry in using deep learning algorithms.
The first achievement was using deep learning for leaf counting as well as implementing
a platform for plant image analysis. The outcome of my work is an automated deep learning model that takes in images of top-down rosette plants and produces a total leaf count. The model achieves state-of-the-art results using relatively easier to obtain annotations than other models proposed by the literature. This work was published in a computer vision conference workshop and formed the basis of further work in this field.
A second achievement was developing a multi-task deep learning model which can identify multiple traits at the same time. There are several benefits of having one model perform several tasks. It makes it easier to manage from a user perspective as well as helps the model learn better if the tasks are correlated to each other. I used this property to demonstrate that by using multi-task learning, the model can be trained with a fraction of the annotations for one of the tasks without significantly impacting overall performance. This work was published in a technical journal.
A third achievement was in deep learning model interpretability. Deep learning produces impressive results for many computer vision applications however they are considered 'black box' models lacking a straightforward explanation of how a network achieves a prediction . In all my works I have included sections on interpretability because it is important to know that the model is learning properly. I have published a paper on this topic focusing on visualization and trying to understand what the networks are focusing on when predicting plant phenotyping traits.
Exploitation Route My research can be continued in both academia and industry. I showed that deep learning can be used successfully for agricultural and plant phenotyping tasks. There are many areas still left to explore. From an academic point of view an interesting PhD topic would be to use deep learning to simulate plant growth from data gathered. Another academic topic which I didn't cover but is relevant to the field is plant health and disease detection which can be another task for the multi-task learning framework. In an industrial setting, the models developed from this award could be used in controlled environment greenhouses to increase the efficiency and yield of crops.
Sectors Agriculture, Food and Drink,Digital/Communication/Information Technologies (including Software),Environment
 
Description I have presented my research during this grant in top conferences and meetings in the UK and the world. During this grant I have become a member of PhenomUK. The organization aims to bring together academics and industry professionals to identify the UK's key phenotyping needs and research and development strategies, driven by technological innovation, which will lead to more efficient integrated plant phenotyping solutions. I have presented part of my research to the PhenomUK annual meeting. This involvement is the first step in translating academic research into applications which have an economic impact. The premise of the research done during the grant was to create algorithms that can increase the quality of life of people working in precision agriculture and plant breeding sector by automating previously manual tasks. The code and applications generated during this grant have been made publicly available.
First Year Of Impact 2019
Sector Agriculture, Food and Drink,Digital/Communication/Information Technologies (including Software)
Impact Types Economic
 
Title Automatic leaf counting model 
Description The model is an automated, deep learning based approach for counting leaves in model rosette plants. While state-of-the-art results on leaf counting with deep learning methods have recently been reported, they obtain the count as a result of leaf segmentation and thus require per-leaf (in-stance) segmentation to train the models (a rather strong annotation). Instead, our method treats leaf counting as a direct regression problem and thus only requires as annotation the total leaf count per plant. Furtherore this model benefits from combining different datasets when training the deep neural network. 
Type Of Material Computer model/algorithm 
Year Produced 2017 
Provided To Others? Yes  
Impact The model was the winner of the computer vision problems in plant phenotyping workshop challenge in 2017. This model was the foundation of several other projects and publications as well as several MSC projects. 
URL https://github.com/andobrescu/Leaf-Counting
 
Title Multi-task model for plant phenotyping. 
Description This model shows how different phenotyping traits can be extracted simultaneously from plant images, using Multi-Task Learning (MTL). MTL leverages information contained in the training images of related tasks to improve overall generalization and learns models with fewer labels. We present a Multi-Task Deep Learning framework for plant phenotyping, able to infer three traits simultaneously: (i) leaf count; (ii) projected leaf area (PLA); and (iii) genotype classification. We adopted a modified pretrained ResNet50 as a feature extractor, trained end-to-end to predict multiple traits. We also leverage MTL to show that through learning from more easily obtainable annotations (such as PLA and genotype) we can predict a better leaf count (harder to obtain annotation). We evaluate our findings on several publicly available datasets of top-view images of Arabidopsis thaliana. 
Type Of Material Computer model/algorithm 
Year Produced 2020 
Provided To Others? Yes  
Impact This work has achieved an improvement on the state-of-the-art for leaf counting compared to direct regression approaches for the datasets tested. The model architecture means that it can compensate for missing labels, leveraging other related traits. When using the model it is not necessary to annotate all the images to maintain performance which can save significant time when trying to implement it outside the lab. 
URL https://github.com/andobrescu/Multi_task_plant_phenotyping
 
Title IDIEL Plant 
Description iDIEL Plant can segment Arabidopsis rosettes from both VIS (light) and NIR (dark) images and can extract quantitative estimates of projected rosette area (PRA). The software automatically numbers plants from left to right as they appear in the image, or manually by clicking on the centre of each plant. Thus, plants could be analysed regardless of arrangement in the image. 
Type Of Technology Software 
Year Produced 2017 
Open Source License? Yes  
Impact This software was used in the University of Edinburgh Institute for molecular plant science.