4D-HSI-4-Free: Integrating Sensors & Vision Process Engineering to Deliver a Tool for revealing 3D Hyperspectral Agri-data from 2D Images

Hyperspectral and multispectral imaging for crop phenotyping and stress analysis has seen significant growth in recent years for wide area scanning, notably from satellites and aircraft. This has been principally based on variants of the Normalised Difference Vegetation Index, 'red edge', ratios and / or chlorophyll fluorescence bands. Though more subtle agri-analysis is being introduced within these remote monitoring technologies, notably with the launch of the ESA Sentinel family of satellites, the reliance on sunlit imaging and the physical engineering limits on resolving power of the optics, alongside atmospheric variations in absorbance and refraction, set hard boundaries on the spatial and spectral sensitivity.

In more recent years Close-Proximity Hyperspectral Imaging (CP-HSI) has been progressed by a number of research groups, in order to ground-truth remotely collected images as well as to gain data with greater subtlety and differentiating power than can be obtained from the remote platforms, due to the physical constraints described. Additionally, CP-HSI offers the potential for greater data availability by avoiding the issues of cloud-cover, and other obstructions, as well as the lack of access to images whilst satellites are orbiting. These CP-HSI studies have tended to adopt the same high-cost passive HSI instruments, as used by the geospatial imaging sector, and then take advantage of the variations in plant and soil data arising intra- and inter-crop canopy to identify the new agri-features.

The e-Agri research team at the University of Manchester have worked over recent years on translating the above concepts into lower-cost and miniaturised engineered systems through replacing the passive HSI instrumentation with active systems, based on broadband proprietary silicon imaging detectors coupled with narrowband LED sources. The resulting Active Close-Proximity (ACP) HSI systems are not only several orders of magnitude cheaper and higher spatial resolution, than their remote passive equivalents, they also offer spectral signal-to-noise that can far exceed what is possible from the passive instruments.

Encouraging as this is, there are engineering challenges associated with active CP-HSI, not least the need to compensate for the orientation affects from position of the biological specimen with respect to the sensor system as well prevent corruption of the spectral data by variations in the lighting location and optical power of the LEDs. These are reconcilable through integrating the optical engineering of multiple sources alongside Photometric Stereo (PS) image reconstruction. The PS technique has arisen as an alternate technique to structured-light for delivering surface normals and texture information at unprecedented spatial resolution, while using inherently low-cost equipment. PS involves capturing a number of images (3+) of an object under varying but known lighting, with each image providing a constraint on the orientation at each pixel. Work pioneered by the CMV has allowed the PS technique to move beyond a laboratory setting towards real-word applications, by relaxing prior assumptions, such as Lambertian surface reflection, collimated illumination and by extending the technique to moving applications.

The team will deliver a new class of 4-Dimensional imaging within an accessible tool which will only marginally increases the component cost versus the 2D hyperspectral precursor. The project is achievable as it will be accelerated via the e-Agri UoM team partnering with the Centre for Machine Vision, at the Bristol Robotics Laboratory, to translate their PS algorithms so that they may be integrated within the ACP-HSI sensors research. The merger of these capabilities will prove the viability of a miniaturised 4D imaging system and deliver a proof-of-principal prototype system for characterisation against plant, animal and aquatic samples.

Hyperspectral and multispectral imaging for crop phenotyping and stress analysis has seen significant growth in recent years for wide area scanning, notably from satellites and aircraft. This has been principally based on variants of the Normalised Difference Vegetation Index, 'red edge', ratios and / or chlorophyll fluorescence bands. Though more subtle agri-analysis is being introduced within these remote monitoring technologies, notably with the launch of the ESA Sentinel family of satellites, the reliance on sunlit imaging and the physical engineering limits on resolving power of the optics, alongside atmospheric variations in absorbance and refraction, set hard boundaries on the spatial and spectral sensitivity.

In more recent years Close-Proximity Hyperspectral Imaging (CP-HSI) has been progressed by a number of research groups, in order to ground-truth remotely collected images as well as to gain data with greater subtlety and differentiating power than can be obtained from the remote platforms, due to the physical constraints described. Additionally, CP-HSI offers the potential for greater data availability by avoiding the issues of cloud-cover, and other obstructions, as well as the lack of access to images whilst satellites are orbiting.

The team will deliver a new class of 4-Dimensional imaging within an accessible tool which will only marginally increases the component cost versus the 2D hyperspectral precursor. The project is achievable as it will be accelerated via the e-Agri UoM team partnering with the Centre for Machine Vision, at the Bristol Robotics Laboratory, to translate their PS algorithms so that they may be integrated within the ACP-HSI sensors research. The merger of these capabilities will prove the viability of a miniaturised 4D imaging system and deliver a proof-of-principal prototype system for characterisation against plant, animal and aquatic samples.

The potential duties, that are actively being pursued by the Manchester team currently fall into four areas. The first is the early detection of fungal disease symptoms on leaf. The initial work in this area was in partnership with the plant scientist at the University of Bonn, and demonstrated that the ACP-HSI approach could be delivered in a package costing euro 100s as opposed to euro 100,000s. This then led on to a small proof of concept project, to show how such a package could exploited alongside a Smartphone handset to enable in-field plant diagnostics and autonomous advice services to be made available in an appropriately costed package for mass deployment to even the poorest smallholder farmers, to enhance their yields through more informed decision making. Incorporating the PS technique in this 'handset package' would add marginal cost but would enable plant scientists access to texture information as well as quantifiable HSI images from what could be 10,000s and then 1Ms of new data-points arising from farmers around the globe, in even the remotest areas. The potential to use 'Big Data' mining approaches on this more rich dataset, by virtue of the PS technique, are significant and have obvious implications on yield forecasting and crop stress mapping that impact on governmental decision making all the way through to mitigating insurance risks.

The second area of activity is in precision intra-row weed detection and eradication. This is due to both the growing resistance, by weeds, to such treatments or the lack of suitable products in the first place (e.g. ryegrass in cereals globally and Amaranthus spp and Conyza spp. in US maize), as well as the removal of older chemistries in a number of regions, notably the EU (ref. directive EU91-414), due to health concerns. The PS approach will become a key element of this autonomous weed identification technique for precision farming as the orientation information it offers will open up the most subtle of classification challenges, such as Blackgrass in post-emergence cereal crops, as the leaf-angle to the sensor will be effectively random.

The third area of ongoing research, that this PS approach will impact, is protein prediction in pre-harvest crops, notably winter wheat. Unlike previous work on late-stage growth or protein measurement in post-harvest grain, the ACP-HSI approach exploit the image data and not the composite NIR spectra. This potentially enables protein development to be tracked at much earlier stages in the growth cycle with corresponding smaller and more precise control of nutrient additions to gain acceptable post-harvest protein content, with the corresponding reduction in farm input costs and carbon-footprint. The PS technique offers the potential to take this tool from a manual technique, where wheat-ear samples are clamped and scanned by a handheld device whilst field-walking, to one that would be applicable to robotic autonomous scanning as it would reduce the impact of ear topology on the HSI data.

The fourth area is in partnership with the Natural Resources Institute (University of Greenwich) the ACP-HSI approach is being investigated to detect insect disease vectors, notably white fly for mosaic virus in cassava. Without PS the technique may be able to detect and, most importantly, speciate the adults (1-2mm) and possibly the nymphs (0.1-1mm), between potential disease carriers and benign insects. However, PS integrated to ACP-HSI offers the potential to reveal the eggs which are hidden within a leaf structure and so determine the full viral infection probability within a crop, and so prevent contamination of the future seasons seed-bank.

Additional applications for the PS technique are also potentially viable within: fish parasite detection, post-harvest produce quality traits in pack-houses (e.g. texture analysis of bagged salads) or soil organic carbon monitoring, amongst other duties.