Video image recognition for ecological monitoring
Lead Research Organisation:
University of Exeter
Department Name: Biosciences
Abstract
Advances in video technology over the last decade mean it is now feasible to study insects and other small animals in their natural habitats. By tagging and genotyping individuals, their lives can be observed and the relationships between individuals, including how many offspring they leave can be quantified. The main bottleneck preventing a range of fantastic new studies is the time it takes to collect basic data from digital video recordings. We will develop new software to automate individual recognition of tagged invertebrates and other small animals in the wild. The project is a collaboration between the School of Biosciences and the School of Engineering, Computing & Mathematics (SECAM) at the University of Exeter. In Biosciences we have developed an 80 camera integrated network of video cameras monitoring a tagged population of wild crickets in a field in Spain. This provides a perfect platform for the development of a system that will have potential applications across field biology. In SECAM we have developed new approaches to object recognition and video tracking that are ideally suited to the requirements of these new monitoring techniques. This technology has the potential to catalyse a host of new studies that will ultimately provide insights into the natural ecology of invertebrates, essential for understanding ecosystems. Potential applications The availability of internet protocol (IP) CCTV cameras, which are flexible to deploy and can be wireless, has enormous potential for use in environmental research. Initially species that predictably visit small areas such as nests, flowers and burrow entrances will be the main subjects of study, but higher resolution cameras enable larger areas to be monitored. Many of our potential end users will not yet be aware of the potential studies they could be carrying out. We anticipate our software will allow behavioural ecologists to monitor the behaviour of beetles, allow pollination studies that automatically track which individual bees have visited which real or artificial flowers, and will allow studies of how the spatial distribution of terrestrial and marine gastropods arises from individual movement patterns. Technology We propose to develop new pattern recognition techniques that will allow any researcher working on a system in which individual animals are tagged to automate key aspects of data collection and analysis. We will use our database of 100,000+ hours of recordings of tagged crickets as a development platform. The crickets have tags with a 2 character code that can be seen on the video recordings. We will use image segmentation via deformable object contours and tracking to automatically distinguish and follow individuals. Variational Bayesian methods will permit efficient handling of multi-modal densities, enabling us to track more than one individual and distinguish animals from noise such as the shadows of grass moving in the breeze. Character recognition will be used to identify individuals from their tags; as these are often not visible or are obliquely viewed, we shall fuse estimates from tracked frames to provide an optimal estimate. We shall also investigate the use of two dimensional error-correcting barcodes (e.g., QR codes) as a robust alternative to alphanumeric codes. By the end of the project we expect the system to be operational in the field and available to researchers for testing and for further development - i.e. at TRL level 4.
Organisations
Publications
Fisher DN
(2016)
Comparing pre- and post-copulatory mate competition using social network analysis in wild crickets.
in Behavioral ecology : official journal of the International Society for Behavioral Ecology
Hopwood PE
(2016)
The effect of size and sex ratio experiences on reproductive competition in Nicrophorus vespilloides burying beetles in the wild.
in Journal of evolutionary biology
| Description | We have worked on the problem of automatically locating, tracking and identifying wild field crickets (Gryllus campestris) in their natural habitat through digital video recordings. Our original aims were to: 1. Detect, count and log the presence of objects of a particular size in our video recordings 2. Make repeated estimates of the characters on a tag providing an estimate of ID. 3. Read tags from more than one individual at once. 4. Track and quantify movement of a single individual. 5. Track and quantify movement of more than one individual at once. 6. Flag novel video segments (e.g. bird devouring a cricket) for manual inspection. 7. Any number of increasingly sophisticated additional features. Of these, the most important aims were the automatic tracking and identification of crickets, with programs written in Matlab. The data are stored in Matlab MAT files and key data extracted to CSV files for importation into a Microsoft Access database in which manually collected data has already been stored. A prototype GUI has been demonstrated which enables the user to view selected videos, with or without overlaid tracks. We have demonstrated a system that is able automatically to detect and log the presence of a moving object within the frame, track and quantify the movement of a single individual, determine to some degree of accuracy that the individual is a cricket, flag some types of unusual events, and make repeated estimates of the characters on a tag. Our tracking algorithm should be able to locate and track more than one object at once (which would allow the reading of more than one tag at once), but we discovered that this was computationally very intensive and the speed of the tracking part of the algorithm needs to be improved. Our work has provided a framework for this development. We have demonstrated that the algorithm can be generalised to other tracking scenarios by using it to locate and track bees in a laboratory environment. The only significant change required was to the prior information about the anticipated size of the object being tracked. Progress against the specific aims of the original NERC proposal is as follows: Detect, count and log the presence of objects of a particular size that move within the frame. The algorithms detect and log the presence of a moving object within each frame and they are able to incorporate a prior belief regarding the size of the object to be tracked. I believe that they are able to track multiple objects at once, but the tracking program is currently too slow to test this functionality to any useful degree. Figure 2 shows an example of a cricket being tracked, compared with the ground truth. The location of the cricket is accurately estimated. Figure 3 shows the tracks made by crickets in a number of videos with the position of the (manually located) burrows marked. The patterns of the tracks are closely associated with the burrows. Make repeated estimates of the characters on a tag providing an integrated best estimate of ID. We have developed an algorithm that uses information from the tracking program to home in on the location of the tag and then compare the extracted tag image with library images of the tags we know to be in use. The algorithm ranks each library tag according to how likely they are to match the extracted tag image. This process can be repeated across all the frames in a video and the resulting ranks used to estimate the ID. If the crickets are being tracked effectively, then it is not necessary to identify them in every frame of a video. In future, the identification process might be performed only on a selected subset frames and it might use information from across the field to inform the estimated tag identity in a particular location at a particular time. Read tags from more than one individual at once. The tag-reading algorithm is able to attempt to locate and read tags from as many objects as the tracking algorithm has tracked. At present this is only single objects. Track and quantify movement of a single individual. The algorithms track and quantify the movement of a single individual. Track and quantify movement of more than one individual at once. As stated above, I believe that they are able to track multiple objects at once, but this has not yet been tested. Flag novel video segments (e.g. bird devouring a cricket) for manual inspection. The cricket tracks are represented by a series of ellipses centred on the cricket. The five parameters that describe each ellipse (xy coordinates of the centre, long and short axis lengths and orientation angle) are estimated as probability distributions. Thus we may identify when the cricket makes an unexpectedly large jump in location or when the size of the ellipse changes unexpectedly. Any number of increasingly sophisticated additional features - in particular the potential to infer behaviour of individuals (e.g. fighting, foraging, mating). Most of the behaviour that we wish to infer is based on the interaction between two or more crickets. Since we are only tracking one individual in each video, we are not able to identify behaviour such as fighting or mating. Foraging and basking behaviour might be plausibly inferred from the shape of the tracks. One particularly useful outcome would be to estimate the location of the burrows. These are often difficult, if not impossible, to see by eye, but should be the focus of large jumps by the (frightened) cricket. Thus multiple, unexpectedly large jumps to a similar location might be used to infer the burrow location. Analysis should be automatic, i.e. the videos should be able to be processed and analysed without human intervention. The processing from video to tracks is entirely automatic. Classification of tracks into cricket and non-cricket, and the location and identification of tags are also automatic and could easily be incorporated into the main process. Cricket tracking data to be uploaded into a database. Rather than an SQL database, the data is stored in Matlab MAT files and key data extracted to CSV files for importation into a Microsoft Access database in which manually collected data has already been stored. Designing and implementing an SQL database would be relatively straightforward. Building a useful front end to it would be more time-consuming, but not too difficult. Long-term maintenance of the front end wold require specialist support on an ad-hoc basis. A graphical user interface (GUI) through which the results could be reviewed. A prototype Matlab-based GUI has been demonstrated which enables the user to view selected videos, with or without overlaid tracks. Other outcomes The tracking algorithm is able to track a range of objects in different formats of video with minimal change. For example, it has been demonstrated on a video of caged bees, with changes only to the video format (an uncompressed AVI rather that the JPEGs for the cricket videos) and the prior expectation of the object size. The research outcomes for computer science are a novel method of probabilistic background modelling (paper in review with Machine Learning) and a novel object tracking algorithm which does away with a separate object (re)aquisition step (paper in preparation). |
| Exploitation Route | We are working with commerical CCTV provider i-code systems to develop their i-catcher software. This will enable users including non-academic researchers, land managers, conservationists, outdoor wildlife attraction managers and so forth to use our software. We are seeking further funding to develop this research. We are working with commerical CCTV provider i-code systems to develop their i-catcher software to enable them to sell it to researchers |
| Sectors | Electronics,Environment,Leisure Activities, including Sports, Recreation and Tourism |
| Description | We are continuing to work on this project with the aim of producing a commercially valuable object recognition and tracking application for biological research. We had 1 PhD student in the UK and 2 masters students in Denmark working on the project. We are now working with a group in the US. However as yet, there have been no documentable impacts |
| Sector | Digital/Communication/Information Technologies (including Software) |
| Impact Types | Economic |