Facial Re-identification Using Deep Learning on Combined Real-Virtual Environments
Lead Research Organisation:
Queen's University Belfast
Department Name: Sch of Electronics, Elec Eng & Comp Sci
Abstract
Face recognition, identification and verification are common problems in the field of biometrics. Many approaches have been developed over the years resulting in robust and accurate methodologies, specially with the arrival of deep neural networks. However, in spite of the ongoing progress, algorithms still suffer when scaling to hundreds/thousands of users with the multiple sources of variability introduced by the environment and the user context, such as head orientation, facial accessories, different backgrounds and the use of different recording devices.
The aim of this work is to address these limitations and to develop a novel unified framework for feature extraction and metric learning using deep learning architectures. By assimilating face recognition to re-identification, we aim to extend the application of our methodology to vast datasets such as current social networks or national police databases, where in most cases only a few images per subject are available (for example their profile picture or police mugshots). The algorithm should be able to use these few photos to identify subjects in CCTV footage at low resolution and image quality in different view-points and poses.
The testing of such a virtually enhanced face re-identification paradigm on real world surveillance cameras will be the underlying objective of this project. The project will aim to develop an approach which can use one or two images from police mugshots to identify individuals in CCTV stream.
The objectives of this work include:
- To implement deep convolutional neural networks to tackle automatic feature extraction.
- To combine automatic feature extraction with metric learning in Siamese network architectures, to specifically address the problem of verification and re-identification.
- To investigate new configurations for face recognition in zero-shot and one-shot scenarios.
- To extend the re-identification framework to allow image-to-video recognition that's able to tackle poor quality CCTV footage.
- To explore robust strategies such as data augmentation and dropout to address facial occlusions and other sources of variation which may affect identification, such as different poses, clothing, cosmetics, glasses, scarfs, hats, etc.
- To develop new deep learning architectures that enhance visual facial re-identification using semantic information from the context, or the subject profile.
- To evaluate the performance and limitation of verification versus re-identification, to link real life snapshots with social network profiles and/or CCTV captures.
This research has potential impact for crime prevention -where mugshots can be matched against CCTV cameras, human trafficking -where pictures in illegal websites can be compared against images taken at the port of entry- and increasing security in airports, -where passport pictures or images in a suspect watch list are compared against the airport cameras, leading to the implementation of more secure and transparent e-gates. This research fits the EPSRC Research areas of Artificial Intelligence Technologies and Image and Vision Computing.
The aim of this work is to address these limitations and to develop a novel unified framework for feature extraction and metric learning using deep learning architectures. By assimilating face recognition to re-identification, we aim to extend the application of our methodology to vast datasets such as current social networks or national police databases, where in most cases only a few images per subject are available (for example their profile picture or police mugshots). The algorithm should be able to use these few photos to identify subjects in CCTV footage at low resolution and image quality in different view-points and poses.
The testing of such a virtually enhanced face re-identification paradigm on real world surveillance cameras will be the underlying objective of this project. The project will aim to develop an approach which can use one or two images from police mugshots to identify individuals in CCTV stream.
The objectives of this work include:
- To implement deep convolutional neural networks to tackle automatic feature extraction.
- To combine automatic feature extraction with metric learning in Siamese network architectures, to specifically address the problem of verification and re-identification.
- To investigate new configurations for face recognition in zero-shot and one-shot scenarios.
- To extend the re-identification framework to allow image-to-video recognition that's able to tackle poor quality CCTV footage.
- To explore robust strategies such as data augmentation and dropout to address facial occlusions and other sources of variation which may affect identification, such as different poses, clothing, cosmetics, glasses, scarfs, hats, etc.
- To develop new deep learning architectures that enhance visual facial re-identification using semantic information from the context, or the subject profile.
- To evaluate the performance and limitation of verification versus re-identification, to link real life snapshots with social network profiles and/or CCTV captures.
This research has potential impact for crime prevention -where mugshots can be matched against CCTV cameras, human trafficking -where pictures in illegal websites can be compared against images taken at the port of entry- and increasing security in airports, -where passport pictures or images in a suspect watch list are compared against the airport cameras, leading to the implementation of more secure and transparent e-gates. This research fits the EPSRC Research areas of Artificial Intelligence Technologies and Image and Vision Computing.
Organisations
Studentship Projects
| Project Reference | Relationship | Related To | Start | End | Student Name |
|---|---|---|---|---|---|
| EP/N509541/1 | 01/10/2016 | 30/09/2021 | |||
| 1941354 | Studentship | EP/N509541/1 | 01/10/2017 | 31/07/2021 | Glen Brown |
| Description | The area we are working in is in facial reidentification. Simply, can we match and image of a face we have (the query / probe) to a list of known faces (the gallery). So far, we have primarily been looking at the data used to train these facial reidentification systems. The datasets which are publicly available that are suitable for training, are primarily harvested from the internet and consist of images of celebrities or politicians. These photographs are usually taken by media professionals with professional-grade equipment. While using these images is an efficient way to obtain datasets that are high-quality and labelled (the process to harvest them can be largely automated), these types of images are not that representative of facial reidentification situation in real-life: for example, images taken from low quality CCTV footage. In our first publication, we showed that the lab performance of facial reidentification systems using machine learning does not necessarily translate to the real world. This is because the type of images we use to train these systems on is not reflective of real-world scenarios. We tested this by creating a dataset based on a more realistic scenario: a workplace where you want to match low-quality CCTV-style footage to a gallery of higher quality employee photographs. In our second publication, we attempted to tackle the problem of only having a relatively small amount of "real-world" data, but trying a novel data augmentation technique. Typically, with Machine Learning and in particular Deep Learning, the more data that's available to train systems on the better. Therefore, if we can use increase the amount of data we have to train with, we should get increased performance. We generated a synthetic dataset of facial images using 3D rendering software, consisting of poses and lighting conditions that are not typically found in the aforementioned datasets. We then used these images in conjunction with the limited amount of real data we had, to train a facial re-identification system that had better performance than when it was just trained on the real facial data. |
| Exploitation Route | By the end of this award we hope to have made a substantial enough contribution to the field of Facial Reidentification (in particular when the amount of training data is considered too small to adequately train a system on), that anyone deploying these types of systems may turn to our work to achieve a boost in performance, even if it is somewhat modest. This would not only impact the engineers and developers directly working on these systems, but also the users of these systems in every sector where facial reidentification may be of use. |
| Sectors | Aerospace, Defence and Marine,Digital/Communication/Information Technologies (including Software),Government, Democracy and Justice,Security and Diplomacy |