BTaRoT: Bayesian Tracking and Reasoning over Time

Lead Research Organisation: University of Sheffield
Department Name: Automatic Control and Systems Eng

Abstract

In this project we will provide new advances in computational methods for reasoning about many objects that evolve in a scene over time. Information about such objects arrives, typically in a real-time data feed, from sensors such as radar, sonar, LIDAR and video. The tracking problem for such scenarios is a well-trodden area, studied for many decades by many researchers. The new and exciting part of this project is in automated understanding of the `social interactions' that underlie a multi-object scene. Can we learn the emerging network structure that develops between objects, in terms of things like who is following who, where is a particular group of objects heading (danger zone or friendly air-field?), has an object left one group and joined another, has a new set of network interactions suddenly come into force? We also seek to integrate this kind of deeper understanding of a complex scene with a simultaneous handling of all of the sensor information available and the decision-making tasks that are required (which sensors to swich on/off, whether an object is friendly or a source of danger, whether an object behaves like a land-rover or a civilian car).

These sophisticated and difficult problems can all be posed very elegantly using probability theory, and in particular using Bayesian theory, a generic inferential and decision-making methodology that allows one to infer hidden information about a system given data from sensors and some prior beliefs about general behaviour patterns of objects. While generic and straightforward to pose, there are substantial challenges for our problem area in terms of how to pose the underlying prior models (what is a good way to model the random behaviour of networked objects in a scene?), and how do we carry out the very demanding computational calculations that are required for many-object scenes? These modelling and computational challenges form a major part of the project, and will require substantial new theoretical and applied algorithm development over the course of the project. We will develop novel computational methods based principally around Monte Carlo computing, in which very carefully designed randomised data are used to approximate very accurately the integrations and optimisations required in the Bayesian approach.

The outcomes from this ambitious project could cause a paradigm shift in tracking methodology if successful, moving away from the traditional viewpoint of a scene in which objects move independently of one another, towards an integrated viewpoint where object interactions are automatically learned and used in improved decision-making processes. We anticipate that the impact will be substantial across a wide range of related disciplines, from ecology and animal behaviour studies through to economic and social networking.

Publications

10 25 50
 
Description The first stage of the research was devoted to modelling interactions between entities in a group of objects and designing sequential Monte Carlo filters. For tracking the motion of the group, box particle filters were developed - for extended and group object tracking. More results are developed for behaviour analysis in video. These are useful for autonomous and semi-autonomous systems. The main software developed has been now made available on the ORDA web site of the University of Sheffield, on GitHub and on ResearchGate.
Exploitation Route Our findings can be used in many different ways - research, industry implementations, ICT applications, security, surveillance, transportation systems and others. The new nachine learning approaches that we developed are opening new avenues, especially related with the autonomous knowledge extraction from data.
Sectors Aerospace, Defence and Marine,Digital/Communication/Information Technologies (including Software),Education,Manufacturing, including Industrial Biotechology,Security and Diplomacy,Transport

 
Description Algorithms and techniques are developed for groups and extended object tracking. Some of these are applied in surveillance systems and intelligent transportation systems. Some of the findings are currently used in a EU funded project, called SETA. It is for prediction of mobility in cities. The results and developed approaches during the BTaRoT project led to new projects where autonomy is a key aspect, e.g. BTaRoT lead to a new knowledge exchange grant in manufacturing, for automated and autonomous recognition of the size of the developed objects and to a new project in on co-botics.
First Year Of Impact 2017
Sector Aerospace, Defence and Marine,Construction,Digital/Communication/Information Technologies (including Software),Education,Environment,Transport
Impact Types Economic

 
Description EU FP7 ITN Marie Curie
Amount € 615,000 (EUR)
Funding ID Grant no. 607400, EU ITN Marie Curie project, TRAX (Training network on tRAcking in compleX sensor systems), http://cordis.europa.eu/project/rcn/109279_en.html 
Organisation European Commission 
Sector Public
Country European Union (EU)
Start 10/2013 
End 10/2017
 
Description SETA: An open, sustainable, ubiquitous data and service for efficient, effective, safe, resilient mobility in metropolitan areas
Amount £339,500 (GBP)
Funding ID 688082 
Organisation European Commission 
Sector Public
Country European Union (EU)
Start 02/2016 
End 02/2019
 
Description TRAX: Tracking Complex Systems, EU FP7 ITN Marie Curie project
Amount € 514,340 (EUR)
Funding ID 607400 
Organisation European Commission 
Sector Public
Country European Union (EU)
Start 10/2013 
End 09/2017
 
Description UK Industry
Amount £111,000 (GBP)
Organisation Selex ES 
Sector Private
Country Italy
Start 09/2013 
End 06/2014
 
Title Algorithms and models for autonomous crowds tracking with box particle filtering and convolution particle filtering 
Description Models and algorithms for autonomous crowds tracking are developed. The algorithms are: a convolutional particle filter, a box particle filter and a sampling importance resampling (SIR) particle filter. The programs are modular and self-contained. The code implements the algorithms presented in the paper: "Autonomous Crowds Tracking with Box Particle Filtering and Convolution Particle Filtering", Automatica, vol. 69, pp. 380-394, July 2016, http://www.sciencedirect.com/science/article/pii/S0005109816300887 . The Convolutional particle filter and the (SIR) particle filter are implemented and compared with the Box particle filter. The Box particle filter requires Intlab software. 
Type Of Material Computer model/algorithm 
Year Produced 2016 
Provided To Others? Yes  
Impact The paper proposes a distributed expectation propagation particle filter able to deal with big data. The data is applicable to object tracking in the presence of a high level of measurement noises including "clutter". The algorithm affords dealing with large volumes of data and to achieve high accuracy. 
URL http://www.sciencedirect.com/science/article/pii/S0005109816300887
 
Title Compressive Sensing Based Design of Sparse Tripole Arrays (Matlab code) 
Description This Matlab code generates the results shown in the paper: Hawes, M.; Liu, W.; Mihaylova, L. Compressive Sensing Based Design of Sparse Tripole Arrays. Sensors 2015, 15, 31056-31068. For the code to run you will need to install the freely available package cvx for use with Matlab. This is available from: http://cvxr.com/cvx/download/ 
Type Of Material Computer model/algorithm 
Year Produced 2016 
Provided To Others? Yes  
Impact The code presents a solution to the problem of designing sparse linear tripole arrays. In such arrays at each antenna location there are three orthogonal dipoles, allowing full measurement of both the horizontal and vertical components of the received waveform. We formulate this problem from the viewpoint of Compressive Sensing (CS). However, unlike for isotropic array elements (single antenna), we now have three complex valued weight coefficients associated with each potential location (due to the three dipoles), which have to be simultaneously minimised. 
 
Title Dynamic Hierarchical Dirichlet Process for Anomaly Detection in Video 
Description This is a source code and synthetic data for dynamic hierarchical Dirichlet process for anomaly detection in video, introduced in O.Isupova, D.Kuzin, L.Mihaylova "Dynamic Hierarchical Dirichlet Process for Abnormal Behaviour Detection in Video" in Proceedings of 19th International Conference of Information Fusion, Heidelberg, Germany, July 2016. In this approach we consider the problem of anomaly detection as extracting typical motion patterns from data by topic modeling methods and detect video clips as abnormal if they have low values of likelihood computed with respect to these extracted typical motion patterns. Learning and inference is performed in a fully unsupervised manner. 
Type Of Material Computer model/algorithm 
Year Produced 2016 
Provided To Others? Yes  
Impact The developed model and algorithms can be used in semi-supervised and unsupervised surveillance systems, including for transportation systems, military and maritime applications and other cases when changes in behaviour needs yo be analysed in an autonomous manner. The code can release the load of human operators and help them in making decisions. 
URL http://ieeexplore.ieee.org/document/7527962/
 
Title Matlab Codes for the Shrinkage based Particle Filter 
Description This is the code for the shrinkage particle filter and the data used forits validation. 
Type Of Material Computer model/algorithm 
Year Produced 2017 
Provided To Others? Yes  
Impact The code provides a solution for dealing with sparse data. It is linked with this publication: https://ieeexplore.ieee.org/document/7888926/ The doi is: https://doi.org/10.15131/shef.data.6127049.v1 
URL https://figshare.com/articles/Matlab_Codes_for_the_Shrinkage_based_Particle_Filter/6127049/1
 
Title Signal Model for Location and Orientation Optimisation for Spatially Stretched Tripole Arrays Based on Compressive Sensing (Matlab code) 
Description This is the model developed for the design of the spatially stretched arrays. The related paper is: M. Hawes, L. Mihaylova, W. Liu, Location and Optimisation Orientation for Spatially Stretched Tripole Arrays Based on Compressed Sensing, IEEE Transactions on Signal Processing, 2017, doi: 10.1109/TSP.2017.2655479 http://ieeexplore.ieee.org/document/7827011/ 
Type Of Material Computer model/algorithm 
Provided To Others? No  
Impact The model can be used in the design of antenna arrays. 
URL https://search.datacite.org/data-centers/bl.shef
 
Title Signal Model for the paper "Bayesian Compressive Sensing Approaches for Direction of Arrival Estimation with Mutual Coupling Effects (Matlab code) 
Description This is a model for direction arrival estimation in sensor arrays, with mutual coupling effects. The related paper is M. Hawes, L. Mihaylova, F. Septier, S. Godsill, Bayesian Compressive Sensing Approaches for Direction of Arrival Estimation with Mutual Coupling Effects, IEEE Transactions on Antennas and Propagation, Vol. 65, No. 3, 2017. 
Type Of Material Computer model/algorithm 
Provided To Others? No  
Impact The model can be used in the design of antenna arrays, and by accounting for the mutual coupling effects between the separate elements. 
URL https://doi.org/10.15131/SHEF.DATA.4535825.V1
 
Description Fraunhofer Institute (FKIE): collaboration via two ITN Marie Curie projects, MC IMPULSE and TRAX 
Organisation Fraunhofer Society
Department Fraunhofer Institute FKIE
Country Germany 
Sector Academic/University 
PI Contribution Joint research collaboration in the areas of sensor data fusion and multiple target tracking. Web link to the current EU ITN Marie Curie project, TRAX: http://cordis.europa.eu/project/rcn/109279_en.html The web link to the completed EU ITN Marie Curie project, called MC IMPULSE: https://mcimpulse.isy.liu.se/
Collaborator Contribution Developments of methods for sensor data fusion and multiple target tracking.
Impact Joint publications as listed below: * M. Schikora, A. Gning, L. Mihaylova, D. Cremers, W. Koch, Box-Particle Hypothesis Density Filter for Multi-Target Tracking, IEEE Transactions on Aerospace and Electronic Systems, Vol. 50, No. 3, July, 2014, DOI:10.1109/TAES.2014.120238, in print. * N. Petrov, M. Ulmke, L. Mihaylova, A. Gning, M. Schikora, M. Wieneke and W. Koch, On the Performance of the Box Particle Filter for Extended Object Tracking Using Laser Data, Proc. from the IEEE Sensor Data Fusion Workshop: Trends, Solutions, Applications, Bonn, Germany, 4-6 Sept. 2012, pp. 19 - 24.
Start Year 2008
 
Description Linkoeping University, Sweden 
Organisation Linkoping University
Country Sweden 
Sector Academic/University 
PI Contribution Joint research in the area of signal processing and novel algorithms for filtering and decision making.
Collaborator Contribution Knowledge exchange
Impact The outcomes from this collaboration related with the two ITN Marie Curie projects (MC IMPULSE that funished) and the current ITN Marie Curie project, called TRAX are in multidisciplinary areas - engineering, ICT, applied mathematics and have applications to intelligent transportation systems, surveillance and other areas.
Start Year 2009
 
Description QinetiQ 
Organisation Qinetiq
Country United Kingdom 
Sector Private 
PI Contribution Results in the areas of sensor data fusion, detection, tracking, reasoning over sensor networks and scene understanding.
Collaborator Contribution Knowledge exchange, real testing examples and data
Impact Publications * H. Bhaskar, L. Mihaylova, S. Maskell, Automatic Human Body Parts Detection Based on Cluster Background Subtraction and Foreground Learning, Neurocomputing. Special Issue on Behaviours in Video, Vol. 100, No. 1, pp. 58-72, 2013. * A. Gning, L. Mihaylova, S. Maskell, S. K. Pang, S. Godsill, Ground Target Group Structure and State Estimation with Particle Filtering, Proc. of the 11th International Conf. on Information Fusion, Cologne, Germany, pp. 1176 -- 1183, 30 June - 3 July 2008. * A. Gning, L. Mihaylova, S. Maskell, S. K. Pang, S. Godsill, Group Object Structure and State Estimation with Evolving Networks and Monte Carlo Methods, IEEE Transactions on Signal Processing, Vol. 59, No. 4, April, 1383 -- 1396, 2011.
Start Year 2006
 
Description Selex ES, UK 
Organisation Selex ES
Country Italy 
Sector Private 
PI Contribution Development of approaches for tracking large groups and related software.
Collaborator Contribution Provision of practical scenarios and testing examples.
Impact * N. Petrov, L. Mihaylova, A. de Freitas, Crowd Tracking with Box Particle Filtering, Proc. of the International Conf. on Information Fusion, Spain, 2014 * N. Petrov, L. Mihaylova, A. Gning, Rectangular Extended Object Tracking with Box Particle Filter Using Dynamic Constraints, Proceedings of the IET Target Tracking and Data Fusion Conference, Liverpool, UK, 30 April, 2014.
Start Year 2011
 
Description Thales (NL) via two EU Initial Training Network Projects, MC Impulse, and now a second project, called TRAX 
Organisation Thales Group
Country France 
Sector Private 
PI Contribution We developed novel algorithms for nonlinear estimation, with various applications such as intelligent transportation systems and surveillance.
Collaborator Contribution Provision of data, practical testing examples and scenarios, knowledge exchange. Web link to the current EU ITN Marie Curie project, TRAX (Training network on tRAcking in compleX sensor systems): http://cordis.europa.eu/project/rcn/109279_en.html Web link to the completed EU ITN Marie Curie project, MC IMPULSE https://mcimpulse.isy.liu.se/
Impact This is a multi-disciplinary collaboration, in the areas of engineering, mathematics, signal processing and ICT.
Start Year 2009
 
Description University of Cambridge 
Organisation University of Cambridge
Country United Kingdom 
Sector Academic/University 
PI Contribution Development of novel sequential Monte Carlo filtering approaches. Joint publications: * L. Mihaylova, A. Carmi, F. Septier, A. Gning, S. K. Pang, S. Godsill, Overview of Sequential Bayesian Monte Carlo Methods for Group and Extended Object Tracking, Digital Signal Processing, February, 2014, Vol. 25, pp. 1-16. * A. Y. Carmi, L. Mihaylova, S. J. Godsill (Editors), Compressed Sensing and Sparse Filtering}, Series Signals and Communication Technology, Springer, Berlin Heidelberg, ISBN 978-3-642-38397-7, 2014. * A. Carmi, L. Mihaylova, A. Gning, P. Gurfil, S. Godsill, Inferring Leadership from Group Dynamics Using Markov Chain Monte Carlo Methods, Chapter in Modeling, Simulation, and Visual Analysis of Crowds}, Eds. S. Ali, K. Nishino, D. Monacha, M. Sha, Springer, pp. 335-357, 2014. * A. Carmi, L. Mihaylova, A. Gning, P. Gurfil, S. Godsill, Markov Chain Monte Carlo Based Autonomous Tracking and Causality Reasoning, Chapter in Advances in Intelligent Signal Processing and Data Mining: Theory and Applications, Eds. P. Georgieva, L. Mihaylova and L. Jain, pp. 7-53, 2013. * A. Gning, L. Mihaylova, S. Maskell, S. K. Pang, S. Godsill, Group Object Structure and State Estimation with Evolving Networks and Monte Carlo Methods, IEEE Transactions on Signal Processing}, Vol. 59, No. 4, April, 1383 - 1396, 2011. * A. Carmi, L. Mihaylova, F. Septier, S. Pang, S. Godsill, MCMC-Based Tracking and Identification of Leaders in Groups, Proceedings of the First IEEE Workshop on Modeling, Simulation and Visual Analysis of Large Crowds, Barcelona, Spain, 6 - 13 November 2011. * A. Gning, L. Mihaylova, S. Maskell, S. K. Pang, S. Godsill, Ground Target Group Structure and State Estimation with Particle Filtering, Proc. of the 11th International Conf. on Information Fusion}, Cologne, Germany, pp. 1176 - 1183, 30 June - 3 July 2008. * A. Gning, L. Mihaylova, S. Maskell, S. K. Pang, S. Godsill, Evolving Networks for Group Object Motion Estimation, Proc. of the Institution of Engineering and Technology (IET) Seminar on Target Tracking and Data Fusion: Algorithms and Applications}, 15-16 April 2008, Birmingham, UK, pp. 99-106.
Collaborator Contribution Joint development of methods for nonlinear filtering, including knowledge exchange.
Impact The outcomes are in multidisciplinary areas of engineering, ICT and mathematics.
Start Year 2007
 
Description University of Lille, Telecomme-Lille, France 
Organisation University of Lille
Country France 
Sector Academic/University 
PI Contribution Development of methodological works on Markov Chain Monte Carlo (MCMC) methods
Collaborator Contribution Francois Septier - bringing knowledge on MCMC methods
Impact * L. Mihaylova, A. Carmi, F. Septier, A. Gning, S. K. Pang, S. Godsill, Overview of Sequential Bayesian Monte Carlo Methods for Group and Extended Object Tracking, Digital Signal Processing, February, 2014, Vol. 25, pp. 1-16. * A. Carmi, L. Mihaylova, F. Septier, S. Pang, S. Godsill, MCMC-Based Tracking and Identification of Leaders in Groups, Proceedings of the First IEEE Workshop on Modeling, Simulation and Visual Analysis of Large Crowds, Barcelona, Spain, 6 - 13 November 2011.
Start Year 2007
 
Title Bayesian Compressive Sensing Approaches for Direction of Arrival Estimation with Mutual Coupling Effects 
Description Matlab code for the signal model used in: M. Hawes, L. Mihaylova, F. Septier and S. Godsill, "Bayesian Compressive Sensing Approaches for Direction of Arrival Estimation with Mutual Coupling Effects" IEEE Transactions on Antennas and Propagation (Volume: 65, Issue: 3, March 2017) https://doi.org/10.1109/TAP.2017.2655013 
Type Of Technology Software 
Year Produced 2017 
Open Source License? Yes  
Impact Code for the design of sparse sensor arrays. 
URL https://figshare.shef.ac.uk/articles/Signal_Model_for_the_paper_Bayesian_Compressive_Sensing_Approac...
 
Title Compressive Sensing Based Design of Sparse Tripole Arrays (Matlab code) 
Description This Matlab code generates the results shown in the paper: Hawes, M.; Liu, W.; Mihaylova, L. Compressive Sensing Based Design of Sparse Tripole Arrays. Sensors 2015, 15, 31056-31068. For the code to run you will need to install the freely available package cvx for use with matlab. This is available from: http://cvxr.com/cvx/download/ 
Type Of Technology Software 
Year Produced 2016 
Open Source License? Yes  
Impact The code has been used for sparse arrays design. 
URL https://figshare.shef.ac.uk/articles/Compressive_Sensing_Based_Design_of_Sparse_Tripole_Arrays_Matla...
 
Title Crowd Tracking with the Box Particle Filter 
Description The software package is for tracking crowds of people. It can be used for surveillance or other purposes. The algorithms developed are general and can be applied in other areas. 
Type Of Technology Software 
Year Produced 2016 
Open Source License? Yes  
Impact This is a MATLAB software package implementing the Box particle filter and the convolution particle filter published in the Automatica, 2016 paper. 
URL http://www.mathworks.com/matlabcentral/fileexchange/55657-crowd-tracking-with-the-box-particle-filte...
 
Title Dynamic Hierarchical Dirichlet Process for Anomaly Detection in Video 
Description This is a source code and synthetic data for dynamic hierarchical Dirichlet process for anomaly detection in video, introduced in O.Isupova, D.Kuzin, L.Mihaylova "Dynamic Hierarchical Dirichlet Process for Abnormal Behaviour Detection in Video" in Proceedings of 19th International Conference of Information Fusion, Heidelberg, Germany, July 2016. In this approach we consider the problem of anomaly detection as extracting typical motion patterns from data by topic modeling methods and detect video clips as abnormal if they have low values of likelihood computed with respect to these extracted typical motion patterns. Learning and inference is performed in a fully unsupervised manner. If you use this code or model please cite the above mentioned paper. 
Type Of Technology Software 
Year Produced 2016 
Open Source License? Yes  
Impact This is the implementation of algorithms for anomaly detection that has been used in autonomous and semi-autonomous systems. 
URL https://figshare.shef.ac.uk/articles/Dynamic_Hierarchical_Dirichlet_Process_for_Anomaly_Detection_in...
 
Title Learning Methods for Dynamic Topic Modeling in Automated Behavior Analysis - Code and Implementation 
Description This is source code for the algorithms presented in the paper "Learning Methods for Dynamic Topic Modeling in Automated Behavior Analysis" by Olga Isupova, Danil Kuzin, Lyudmila Mihaylova. Published in IEEE Transactions on Neural Networks and Learning Systems, 2017. DOI: 10.1109/TNNLS.2017.2735364. Two learning methods for the Markov Clustering Topic Model (MCTM) are developed - Expectation-Maximisation (EM) algorithm and Variational Bayes (VB) inference. Implementation is done in Matlab. 
Type Of Technology Software 
Year Produced 2018 
Open Source License? Yes  
Impact The software led to the development of new behaviour analysies approaches. 
URL https://figshare.shef.ac.uk/articles/Code_for_Learning_methods_for_dynamic_topic_modeling_in_automat...
 
Title MATLAB Implementation of a Box Particle Filter for tracking multiple extended objects 
Description This is a software MATLAB Implementation of a Box Particle Filter for tracking multiple extended objects for tracking multiple extended objects. The algorithm is a Box particle filter. The program is modular and self-contained. The code is linked with the paper: "A Box Particle Filter Method for Tracking Multiple Extended Objects", doi: 10.1109/TAES.2018.2874147. The Box particle filter requires a library called Intlab for operation. 
Type Of Technology Software 
Year Produced 2019 
Open Source License? Yes  
Impact The developed approach raised interest in ONERA and their navigation platforms. 
URL https://figshare.shef.ac.uk/articles/MATLAB_Implementation_of_a_Box_Particle_Filter_for_tracking_mul...
 
Title Matlab Codes for the Shrinkage Particle Filter Dealing with Sparse Data 
Description Matlab Codes for the Shrinkage based Particle Filter Fileset posted on 25.04.2018, 17:12 by Aroland Kiring Lyudmila Mihaylova Jose Esnaola It contains the Matlab codes for the developed shrinkage based particle filter. The shrinkage based particle filter combined the shrinkage estimator and the particle filter to jointly estimate the shadowing noise covariance matrix of measurements and the state of the mobile user. 
Type Of Technology Software 
Year Produced 2017 
Open Source License? Yes  
Impact The software is useful to estimate the unknown measurement covariance matrix, for tracking and localisation tasks. 
URL https://figshare.shef.ac.uk/articles/Matlab_Codes_for_the_Shrinkage_based_Particle_Filter/6127049/1