An affordable stereoscopic camera array system for capturing real-time 3D responses to vegetation dense environments

Lead Research Organisation: University of Edinburgh
Department Name: Sch of Biological Sciences

Abstract

To feed our growing global population, the productivity of staple crops will require an increase in yields of ca. 60% by 2050. Realisation of this significant challenge is currently hampered by insufficient capacity of the plant science and Agri-Tech communities to analyse existing plant genetic resources for their interaction with the environment. Plant phenotyping is an emerging science that links genomics with plant ecophysiology and agronomy. The collection of architectural traits of a plant throughout the life cycle (the phenotype) is a result of dynamic interactions between the genetic background (the genotype) and the physical world in which the plant develops (the environment). These interactions determine plant performance and productivity measured as accumulated biomass and commercial yield, and resource use efficiency. A critical parameter of a plant's phenotype is the shape in three dimensions - the plant architecture - which reflects the adaption of a plant to environmental conditions, such as light quantity and quality, and temperature. We will bring together plant scientists, and image capture and machine vision experts to develop an affordable and user-friendly system for progressing on one of the most important issues in biology, our capacity to enhance the productivity of plants. We will combine a robust photometric stereo approach with software using optimised learning algorithms that will produce standardised outputs that can be readily integrated across scales with other biological data sets.

Technical Summary

There is an urgent need for new technologies that allow us to monitor and predict the impact of abiotic and biotic stresses on seasonal plant growth. Such tools could provide quantitative data on crop yield traits and plant fitness markers in natural habitats. The current project is a collaboration between plant researchers, modellers, engineers and image analysis specialists that aims to take a new approach, by producing i) an affordable and robust photometric stereo platform for 3D capture of rosette growth in Arabidopsis that can be extended to other crops, such as related Brassica species (e.g. Brassica rapa), and ii) a standardised open source software based in optimised machine learning algorithms and fronted by a biologist-friendly user interface for simplifying the extraction of traits of interest. To test the system we will target an important plant adaptive response that can negatively impact plant biomass and crop yield - the shade avoidance response (SAR). Initial computational work will focus on processing image data to gather 3D information for plant reconstruction through intelligent integration over the surface normal field. Subsequent image/model analysis will centre on the optimisation of leaf segmentation algorithms using features commonly used in machine vision and new approaches rooted in unsupervised feature learning. Following segmentation, we will quantify the plant phenotypic traits required for evaluating the key features of SAR, including: leaf emergence rates, positioning, orientation, as well as alterations in leaf length, width, shape and expansion. The novel algorithms resulting from this research will be integrated into the existing UoE PhenoTiki package (http://phenotiki.com) so that we reach a wide base of users. We aim to combine 3D phenotype data with mathematical modelling to predict growth outcomes in differing growth scenarios and in unseasonal weather, including controlled environments and outside in field or natural habitats.

Planned Impact

Who will benefit from this research?
1. Academics and researchers in all fields of plant research and machine learning.
2. UK and international science base.
4. The Agro-Tech industry including biotechnologists and plant breeders seeking to increase plant productivity, and metabolic engineers.
5. Agricultural community and advisors.
6. The University of Edinburgh.
6. The postdoctoral researcher (PDRA), and research associate (RA).
7. Public.

How will they benefit?
1. The research will have a major impact on understanding of light signalling and shade avoidance, and enhance the UK's international standing in plant science. Currently no other equipment can perform these measurements elsewhere in the world, nor integrate them with the wealth of molecular data and integrated modelling expertise available at UoE.
2. The technology is designed to enable labs in universities, research institutes and the Agro-Tech industry to pursue phenotyping approaches that have previously been largely inaccessible to them. To acquire the technology following publication, researchers will access an already established open source online resource (PhenoTiki) with an already existing user community to download a user-friendly software and have the option to reproduce the phenotyping system in their own labs, using our published designs that where possible will be made available for 3D printing.
3. Our system will provide a means of standardisation across the phenotyping field, to inform research strategies towards enhanced food security, which will be a platform for direct translation to achieving this goal in food crop plants through yield improvements. The public interest in food security and sustainable crop production, the broad biological scope of the project, and the predictive, systems approach already highlight areas of Impact.
4. The PDRA and RA will receive a wide training in plant integrative biology, hardware design and machine vision approaches, full access to professional skills and wider training courses, and the opportunity to work with two different research institutes.

What will be done to ensure they benefit from this research?
1. Publish results in high-impact journals in a timely fashion, with open access.
2. Present research results at UK and international meetings and institutions.
3. Exploit existing contacts with other UK and international academics with relevant research interests as soon as any exploitable results/materials are generated.
4. Make informal contacts with industrialists as soon as exploitable results/materials are generated; recognise and protect intellectual property to ensure wise and fruitful exploitation.
5. Provide PDRA and RA training and mentoring available at UoE and CMV, including regular reviews to monitor progress and a career development plan. We will also encourage participation in all aspects of the dissemination of research results, and understanding of the wider implications and applications of the research.
7. Use results as part of our regular engagement with non-academic audiences, e.g. local interest groups, schools, local and national shows, science showcases, media.

Publications

10 25 50
 
Description Towards WP1 (Project management): Project management is ongoing by McCormick

Towards WP2 (Rig design & construction) (100% complete): The PS rigs were completed on schedule. Rig 1 was improved with the addition of an NiR filter that substantially improved image capture. Due to the successful operation of the rigs, an additional rig was built in Sept 2017 to accelerate data generation.

Towards WP3 (Data capture) (90% complete): Data capture is complete for validation experiments and a near complete for a matrix" experiment designed to examine the performance of the rigs to capture Arabidopsis growth under different environmental conditions.

Towards WP4 (Computer Vision algorithm design) (85% complete): Bristol and UoE are currently labelling image files to train an rNN network using 3D image data. Preliminary outcomes are promising for tracking individual leaf movements.

Towards WP5 (Analysis of 3D dynamic growth) (80%): Area and rosette shape analysis of the "matrix" experiment is near complete. Work on evaluating individual leaf movement is ongoing with an expected completion by the end of the project.

Towards WP6 (Software integration) (70%): Software for extracting 3D traits is currently command line based (Python), development of a user friendly GUI interface is ongoing.

Towards WP7 (Dissemination): The project work has been disseminated at two conferences (IAMPS and CVPPP-ICCV). A manuscript is being compiled that describes the PS system and demonstrating novel biological data outputs in the matrix experiment.
Exploitation Route The technology shows promise in terms of the data generated thus far. We foresee that this technology will have an impact in the plant science and agriculture sectors. Furthermore, due to the low-cost of the technology, we envisage an impact in the education sector.
Sectors Agriculture, Food and Drink,Education

URL http://mccormick.bio.ed.ac.uk/
 
Description 2017: publication of project related work in a high impact factor journal (Plant Methods) has resulted in new open source software developed for public access (https://datashare.is.ed.ac.uk/handle/10283/2922). 2017-2018 Attendance at workshops at ILRI-BecA (Kenya) and the IITA (Zambia) has helped knowledge exchange of plant phenotyping with agronomists and breeders in Africa, who could utilise the low-cost tools developed in this project. 2017: We successfully applied for funding from Phyconet to further develop a virtual reality outreach tool for the public to teach the public about the importance and shortcomings of Rubisco, the primary carboxylating enzyme in photosynthetic organisms. We have recently used the system at the 2018 Edinburgh Festival of Learning event. 2017-2018: Work on this project has led to a funded PhD students joining the lab in 2018 (EastBio scholarship) and co-supervision of PhD student with Sotirios Tsaftaris (School of Eng).
First Year Of Impact 2017
Sector Agriculture, Food and Drink,Digital/Communication/Information Technologies (including Software)
Impact Types Cultural,Societal,Economic

 
Description GCRF One Planet Workshop at ILRI-BecA (Kenya).
Geographic Reach Africa 
Policy Influence Type Participation in a advisory committee
 
Description Practical Synthetic Biology Workshop (South Africa) - BBSRC call
Geographic Reach Africa 
Policy Influence Type Participation in a advisory committee
 
Title iDIEL Plant software and image capture system design. 
Description Software and source code for a "Do-It-Yourself" phenotyping system: measuring growth and morphology throughout the diel cycle in rosette shaped plants. Plant Methods 13:95. 
Type Of Material Biological samples 
Year Produced 2017 
Provided To Others? Yes  
Impact The outcomes of this research initiated several discussions with potential GCRF collaborators in Africa for technology transfer. 
URL http://mccormick.bio.ed.ac.uk/software
 
Description Stereoscopic camera array system - BRL, Centre for Machine Vision - UWE Bristol 
Organisation University of the West of England
Department Psychology
Country United Kingdom 
Sector Academic/University 
PI Contribution Currently collaborating on the construction and software development of a stereoscopic camera array system for capturing real-time 3D images of plants with the BRL, Centre for Machine Vision (CMV) (Smith Lab).
Collaborator Contribution The CMV has designed and built the rigs, and delivered them to Edinburgh for testing. We collaborate closely with the CMV for building software to extract growth traits.
Impact The project commenced in Oct 2016. Thus far the rigs have been assembled and data is currently being generated and analysed. This is a multidisciplinary collaboration between computer scientists (CMV), biologists and engineers (Ed).
Start Year 2016
 
Description Stereoscopic camera array system - Edinburgh, Eng 
Organisation University of Edinburgh
Department School of Biological Sciences
Country United Kingdom 
Sector Academic/University 
PI Contribution Currently collaborating on the construction and software development of a stereoscopic camera array system for capturing real-time 3D images of plants with UoE School of Engineering (Tsaftaris group).
Collaborator Contribution The Tsaftaris group brings specific expertise in machine learning and algorithm development for Arabidopsis plants. McCormick co-supervises a PhD student in the Tsaftaris group. We have shared access of data, protocols and biological materials between labs.
Impact The project commenced in Oct 2016. Thus far the rigs have been assembled and data is currently being generated and analysed. This is a multidisciplinary collaboration between computer scientists (CMV), biologists and engineers (Ed).
Start Year 2016
 
Description Stereoscopic camera array system - Edinburgh, SBS 
Organisation University of Edinburgh
Country United Kingdom 
Sector Academic/University 
PI Contribution Currently collaborating on the construction and software development of a stereoscopic camera array system for capturing real-time 3D images of plants with UoE School of Engineering (Tsaftaris group).
Collaborator Contribution The Halliday group brings specific expertise in shade avoidance Arabidopsis mutants (Phy mutants) - the primary targets for data generation in this project. McCormick co-supervises a PhD student in the Halliday group. We have shared access of data, protocols and biological materials between labs.
Impact The project commenced in Oct 2016. Thus far the rigs have been assembled and data is currently being generated and analysed. This is a multidisciplinary collaboration between computer scientists (CMV), biologists and engineers (Ed).
Start Year 2015
 
Title Plant Magic software for leaf rosette area analysis 
Description Plant Magic is a software tool for analysing images of plants and calculating the leaf/rosette area under both night and day conditions. An associated hardware tool is used for night imaging. 
Type Of Technology New/Improved Technique/Technology 
Year Produced 2016 
Impact The software was developed by a Masters student (Andrei Dobrescu). Andrei has now been accepted as a PhD student co-supervised by McCormick and Sotirios Tsaftaris (UoE, Eng) to continue work on plant image analysis. The software is currently used by several labs in UoE and will shortly be released on the open Plant Image Analysis website (http://www.plant-image-analysis.org/). An additional hardware tool (a low cost far-red LED array with a raspberry Pi camera) was developed to capture images throughout the diel cycle. This method for building this setup is being prepared for publication. 
 
Description Festival of Learning - virtual reality Rubisco 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Undergraduate students
Results and Impact We applied our outreach tool developed in 2017 to reach a wider audience through the UoE Festival of Learning week. The virtual reality game teaches about the importance and shortcomings of Rubisco, the primary carboxylating enzyme in photosynthetic organisms. Gamification of Rubisco catalysis allowed engagement with the public on the importance of Rubisco and to discuss relatively complex biochemistry.
Year(s) Of Engagement Activity 2017,2018
URL http://www.festivalofcreativelearning.ed.ac.uk/event/photosynthesis-virtual-reality
 
Description GCRF One Planet Workshop at ILRI-BecA (Kenya). 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Attended the GCRF One Planet Workshop at ILRI BecA in Kenya. This was an international meeting attended be researcher and business leaders from over ten countries to discuss opportunities for high impact research collaboration.
Year(s) Of Engagement Activity 2017
 
Description International Institute of Tropical Agriculture (IITA) workshop (Zambia) 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Soybean is a principal food crop in sub-Saharan African countries, such as Zambia, Malawi and Mozambique. Current soybean varieties consistently underperform, which continues to impact negatively on sustainable food security. We discussed potential GCRF-associated collaboration potential with the IITA for adapting the digital imaging tool developed in BB/N02334X/1 to provide a step change in precision breeding capacity in the Zambian soybean breeding community.
Year(s) Of Engagement Activity 2018
 
Description Midlothian Science Festival - virtual reality Rubisco  
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact We developed a new, high-impact educational tool and participated in outreach for young school children (4-12) and their parents during the Scotland Midlothian science festival (http://midlothiansciencefestival.com/). Their objective was to teach the public about the importance and shortcomings of Rubisco, the primary carboxylating enzyme in photosynthetic organisms. Gamification of Rubisco catalysis allowed engagement with the public on the importance of Rubisco and to discuss relatively complex biochemistry, even with four year olds!
Year(s) Of Engagement Activity 2016
URL http://mccormick.bio.ed.ac.uk/amccormi/sites/sbsweb2.bio.ed.ac.uk.amccormi/files/MSF%202016%20Collag...