Wide area multi-target tracking over multiband imaging sensor networks for military applications

This PhD project aims to develop the first wide-area multi-target tracking system for military environments. This system will allow detecting, tracking and monitoring multiple simultaneous targets moving across a network of non-overlapping and diverse vision sensors, including infrared and RGB, on the ground and/or on UAVs. This goal is achieved by proposing a holistic tracking framework based on the Deep Learning Paradigm. Our approach is composed of a multimodal re-identification algorithm, able to preserve the identity of the targets when moving across sensors, and a within-camera multi-target tracking algorithm, which are combined in a unified Deep Neural Network architecture.
Technological advances in sensing have fully transformed military operations, allowing remote monitoring and intervention while reducing the dangers for pilots and soldiers. In this context, target detection and tracking is a fundamental task required for tactical operations and reconnaissance. The increasing ubiquity of sensors on the field increases not only the coverage but its success at the cost of increasing the resources to process the large data flow.
The automatisation of target tracking allows addressing efficiently this scenario while overcoming human limitations. Most existing research in the field assumes tracking is confined within a single sensor's field of view, rather than addressing the complexity of tracking over a network of non-overlapping sensors. This not only reduced coverage but also assumes no interruption of visual contact happens during the duration of the mission, resulting on increasing chances of losing a critical target. In this PhD project, we aim to tackle the previous limitations of the state of the art by proposing a holistic view of tracking, where target detection, multi target tracking and target re-identification across sensors are solved together in a unified framework. The scenario to be solved comprises multiple objects being observed and transiting between multiple non-overlapping cameras, both infrared and RGB, mounted on the ground and/or on UAVs.
To achieve this goal, we propose a unified wide area tracking framework based on Deep Neural networks. This framework will be developed in 3 work-packages:
1. Multi-band re-identification algorithm: In this work-package we will develop a re-identification system able to preserve the identity of the targets when they move across non overlapping sensors, as well as reappearances after long occlusions. The proposed system is based on Deep learning Siamese architectures that enable the join optimisation of feature extraction and metric learning for this task. Given the particularities of military scenarios, the network will be trained using different combinations of RGB, infrared and motion channels in order to avoid over-dependency on visual clues and allow the re-identification when only partial information is given by the sensor.
2. Multi target tracking based on LSTM and recurrent networks: This work-package proposes the novel use of Deep Learning for multi-target tracking for solving the inherent data association and optimisation problem involved in multi-target tracking. Very preliminary attempts to tracking using neural networks have been recently proposed in the literature, but limited to single-target tracking or using simulated results. We aim to extend those approaches to an effective multi-target tracking system.
3. Wide area tracking framework: In this work-package, the previous deliverables will be combined in a unique Deep Neural architecture. This will be possible thanks to the use of neural networks in both components that can be here joined into a unified architecture. This will allow an end-to-end learning paradigm that will maximise the performance of all the components. The result of this work-package will be the first proposed wide-area framework based on deep learning.