Dynamic modelling of plant growth with computer vision

Computer vision as a discipline has recently made great progress in the detection and modelling of articulated shapes (for example, human pose detectors). There is, however, very little work in computer vision on the modelling of organisms which change and grow, like plants. This project aims to change that.

This project will develop computer vision techniques for the modelling, measurement and tracking of plants during growth, working at the level of individual leaves. This is a hard computer vision problem, with application to computational biology, phenomics, and precision agriculture: it involves being able to first find the plants; then detect individual leaves; and then work out what leaves are hidden behind other leaves, and finally to estimate their size and shape of both visible leaves and hidden leaves.

This will be achieved through the use of computer vision to build up a model of the plant as it grows, recorded in timelapse photographs. This model will encode each leaf's size and shape, and whether that leaf is occluded by or occludes any other leaves. This is the first proposal aiming to treat plants as complex, self-occluding objects whilst modelling from a single viewpoint. From the outputs of the software it will then be possible to derive accurate estimates of biomass, leaf area, and other biologically significant measurements.

As well as the software, this project will publish a dataset which will contain photographs of plants (top down images from a single camera), and for certain images it will also contain a labeling, showing which part of the image (which pixel) corresponds to which leaf. The dataset will also include information from plants that have been dissected, such as the areas of individual leaves, and the weight of the whole plant. This dataset will make a significant contribution towards computer vision for plant science by making a standard against which this and all future projects in the area can be judged - there are no other public datasets for this kind of research.

The chosen plant (Arabidopsis Thaliana), is used by biologists as a model organism. This plant is of particular use to the current research project for two reasons: firstly it grows quickly so a full life cycle can be captured on time lapse in a matter of months; and secondly, biologists are interested in precise measurements of the way this plant grows, as its genome has been completely mapped, making it a common object of study in phenomics. Phenomics is the study of the way in which genes and environment interact to produce an organism - a phenome; this involves monitoring precisely the way an organism grows in or reacts to specific environmental situations (phenotyping). Within plant phenomics, monitoring has traditionally been through destructive measures (dissection) or hand measurement; this has clear associated costs both in terms of time and in terms of plants. The relation to phenomics, and in particular the UK's National Plant Phenomics Centre, mean that research in this project has a clear target audience (plant biologists). It is anticipated that in future, the kinds of software and algorithms devised during this project will be extended to other varieties of plants, and will become more widespread, contributing to plant monitoring systems for broader agricultural or even hobbyist gardener usage.

The beneficiaries of this research will be, in the immediate future, academics within computer science and biology. Within computer science the research described in this proposal follows two main avenues of research: occlusion reasoning, and the modeling of growth with morphological change. This research will also provide new techniques within biology, which in turn will provide access to more detailed information about he growth of plants from a dynamic perspective. The main pathways to this academic impact come from the dataset proposed in WP1 and the workshop proposed during WP5.

Looking at the pipeline for future scientific research and researchers, this proposal also has an explicit public engagement element aimed at the next generation of scientists: building a workshop to take into schools. This workshop will involve secondary school students in practical activities that illustrate the science developed during this work, emphasising the ways in which computer science can help plant science, and in which plant science can provide interesting and challenging test cases for computer science. The workshop will be created, trialled in local schools, and then released as a "workshop in a box" enabling broader uptake across the computer vision and plant science communities.

In the medium term, the ability to model and measure plant growth in the field, using time-lapse photography, will benefit a very broad range of people. Accurate, cost-effective measurement of plant characteristics - particularly dynamic plant characteristics - will enable better plant breeding and a better understanding of the relationship between plant genetics, growing environment, and plant health. Algorithms and software such as those developed as part of this research proposal are a key step on the way to fully automated plant monitoring. This is of interest to plant breeders, the food industry, and agriculture in general. It is even of interest to (and has impact upon) hobbyist plant growers: if we can monitor plant growth for phenomics and breeding purposes, we can also monitor plant growth to determine whether the watering regime is right in someone's greenhouse.

Looking further into the future (10-50 years), food security is a key problem of national and international importance: being able to breed for dynamic plant characteristics (things like rapid establishment, growth, and drought resilience) and being able to measure these characteristics (accurately, with high-throughput methods like computer vision) is absolutely vital. This is of direct relevance to the question of where our future meals will come from - predictions of population growth suggest that globally, we need to become more efficient in food production simply to maintain our current levels of food consumption. In addition to food-related impact, there are energy-related impacts for computer vision in plant science: biofuels could solve energy generation problems, and phenomics is addressing some key questions in this domain too.