Face identification within realistic contexts
Lead Research Organisation:
University of Kent
Department Name: Sch of Psychology
Abstract
In many situations, security relies on accurate person identification. At airports, routine person identifications are based on face matching, in which passport officers have to decide whether the face photograph in a presented identity document actually depicts its bearer. People seeking to avoid detection at such security controls may attempt to use illicit ID. Passports themselves are now highly sophisticated documents, which are difficult to forge. For this reason, fraudsters typically use valid identity documents, often belonging to someone with a similar facial appearance. A substantial body of laboratory research now exists to demonstrate that the detection of such impostors - or identity mismatches - is difficult and highly error-prone, even by trained professionals.
However, all existing research has investigated face matching under fairly basic experimental conditions, often presenting pairs of cropped faces on plain backgrounds, or in highly simplified field settings. There is some value in these lab-based studies, because they allow one to study such important variables as individual differences, image quality and so forth. However, these approaches offer a weak proxy for studying how the environment and social interaction affect this task, which provide many additional factors that can affect face matching. For example, passport officers may resort to information from body language to support identification decisions, but environmental factors such as queues and crowd dynamics might also exert pressure on passport officers that could compromise their performance.
The impact of such factors is likely to be very large, as suggested by research on human performance in other realistic settings. However, the sensitive and security-critical nature of passport control makes it practically impossible to study in real life. In this project, we propose a novel approach to investigate these factors, by examining face matching in virtual reality (VR), which can provide the detailed interactive environments that conventional laboratory experiments cannot. This approach is completely new in face matching and allows for the study of factors that may affect person identification in the real world but cannot be captured by laboratory experiments.
We propose a series of three validation experiments to establish that a virtual environment captures the key visual aspects required for person identification. This is followed by eleven main experiments which will test how environmental and social interaction factors affect forensic person identification in a realistic context. These experiments will examine the impact of factors such as a person's body language, passenger queues, and crowd dynamics.
The knowledge gained from these experiments will considerably advance our theoretical understanding of face matching, which currently cannot accommodate the important effects of environmental and social interaction factors on this task. The project will also yield findings of applied importance, for example, in terms of understanding how security in person identification could be enhanced at airports.
In addition, the project will produce a free database of avatars for psychological research on person identification in VR, as well as an open access start-up guide for other researchers with an interest in conducting such research. These resources will create a lasting impact on the development of VR research in Psychology beyond the scope of this project.
However, all existing research has investigated face matching under fairly basic experimental conditions, often presenting pairs of cropped faces on plain backgrounds, or in highly simplified field settings. There is some value in these lab-based studies, because they allow one to study such important variables as individual differences, image quality and so forth. However, these approaches offer a weak proxy for studying how the environment and social interaction affect this task, which provide many additional factors that can affect face matching. For example, passport officers may resort to information from body language to support identification decisions, but environmental factors such as queues and crowd dynamics might also exert pressure on passport officers that could compromise their performance.
The impact of such factors is likely to be very large, as suggested by research on human performance in other realistic settings. However, the sensitive and security-critical nature of passport control makes it practically impossible to study in real life. In this project, we propose a novel approach to investigate these factors, by examining face matching in virtual reality (VR), which can provide the detailed interactive environments that conventional laboratory experiments cannot. This approach is completely new in face matching and allows for the study of factors that may affect person identification in the real world but cannot be captured by laboratory experiments.
We propose a series of three validation experiments to establish that a virtual environment captures the key visual aspects required for person identification. This is followed by eleven main experiments which will test how environmental and social interaction factors affect forensic person identification in a realistic context. These experiments will examine the impact of factors such as a person's body language, passenger queues, and crowd dynamics.
The knowledge gained from these experiments will considerably advance our theoretical understanding of face matching, which currently cannot accommodate the important effects of environmental and social interaction factors on this task. The project will also yield findings of applied importance, for example, in terms of understanding how security in person identification could be enhanced at airports.
In addition, the project will produce a free database of avatars for psychological research on person identification in VR, as well as an open access start-up guide for other researchers with an interest in conducting such research. These resources will create a lasting impact on the development of VR research in Psychology beyond the scope of this project.
Planned Impact
Impact of the project will be achieved through four pathways:
1) Dissemination through academic channels
We will produce a minimum of 7 publications. These will comprise 5 empirical outputs suiting high-impact journals, such as Psychological Science and JEP:General; a start-up guide for researchers on how to run experiments in VR, aimed at Behavior Research Methods as an Open Access article; and a database of the person avatars for this project, which we will make freely available via an Open Access journal such as i-Perception. We have also planned eight conference presentations of this work at national and international meetings of the EPS, BPS, ECVP, ESCoP, and VSS (see project timetable).
2) Targeting of non-academic beneficiaries
Non-academic beneficiaries of this project will be organisations for public safekeeping, such as Border Force UK and the Police. We have long-standing links with these organisations, which we will use to create an advisory group of key Border Force and Police personnel, comprising of senior forensic facial examiners. We will meet with this group 9 months into the project to present our aims and groundwork and to demonstrate the VR environment. The aim here is to receive feedback from this user community early in the project (i.e., before the major data collection is conducted), with a view to tuning VR environment and design of future studies to ensure maximum benefit from our research for these beneficiaries. We will then hold a 2nd meeting at 30 months to report the project findings to this end-user community. The knowledge generated by the project will inform these professionals of which factors undermine person identification at airports and how this can be enhanced. We will also consult these contacts on how to develop our VR airport paradigms into a personnel selection tool for the user community, to recruit individuals with high face-matching ability for relevant roles, as well as a training tool for existing staff. We provide letters of support from Border Force and the Police as attachments.
3) VR training for researchers
Technology has leapt ahead to provide affordable, high-capability VR equipment, but accessibility for academics is still limited, as VR requires a completely different approach to running laboratory experiments. We aim to open up this technology to other researchers by providing (i) regular updates on a dedicated project website, (ii) video demonstrations for setting up experiments in VR on a dedicated project YouTube channel, (iii) a Facebook forum for Psychologists interested in VR for knowledge exchange, (iv) a start-up guide to experiments in VR in an Open Access journal, and (v) a paper detailing the avatar database, which will be made freely available for download. These activities will create lasting positive impact on the development of VR research in Psychology beyond the scope of this project. We have a good track record on delivering these types of materials. For example, Burton's previous ESRC award focused on theoretical developments in face recognition, but he also published a graphical face-manipulation environment (InterFace) as a side-benefit, which is freely available and now used by researchers worldwide.
4) Engagement of wider public
Psychological research has great intuitive appeal for members of the general public, who are also the ultimate indirect beneficiaries of the knowledge developed in this project, through improvements in airport and border security. VR is also rapidly gaining popularity in non-academic contexts, such as computer gaming. To engage these beneficiaries, we will present our research at public science events, such as the British Science Festival, British Science Week, and 'Pint of Science' festivals. In addition, news from this project will be disseminated via UoK and UoY's Psychology Twitter accounts.
1) Dissemination through academic channels
We will produce a minimum of 7 publications. These will comprise 5 empirical outputs suiting high-impact journals, such as Psychological Science and JEP:General; a start-up guide for researchers on how to run experiments in VR, aimed at Behavior Research Methods as an Open Access article; and a database of the person avatars for this project, which we will make freely available via an Open Access journal such as i-Perception. We have also planned eight conference presentations of this work at national and international meetings of the EPS, BPS, ECVP, ESCoP, and VSS (see project timetable).
2) Targeting of non-academic beneficiaries
Non-academic beneficiaries of this project will be organisations for public safekeeping, such as Border Force UK and the Police. We have long-standing links with these organisations, which we will use to create an advisory group of key Border Force and Police personnel, comprising of senior forensic facial examiners. We will meet with this group 9 months into the project to present our aims and groundwork and to demonstrate the VR environment. The aim here is to receive feedback from this user community early in the project (i.e., before the major data collection is conducted), with a view to tuning VR environment and design of future studies to ensure maximum benefit from our research for these beneficiaries. We will then hold a 2nd meeting at 30 months to report the project findings to this end-user community. The knowledge generated by the project will inform these professionals of which factors undermine person identification at airports and how this can be enhanced. We will also consult these contacts on how to develop our VR airport paradigms into a personnel selection tool for the user community, to recruit individuals with high face-matching ability for relevant roles, as well as a training tool for existing staff. We provide letters of support from Border Force and the Police as attachments.
3) VR training for researchers
Technology has leapt ahead to provide affordable, high-capability VR equipment, but accessibility for academics is still limited, as VR requires a completely different approach to running laboratory experiments. We aim to open up this technology to other researchers by providing (i) regular updates on a dedicated project website, (ii) video demonstrations for setting up experiments in VR on a dedicated project YouTube channel, (iii) a Facebook forum for Psychologists interested in VR for knowledge exchange, (iv) a start-up guide to experiments in VR in an Open Access journal, and (v) a paper detailing the avatar database, which will be made freely available for download. These activities will create lasting positive impact on the development of VR research in Psychology beyond the scope of this project. We have a good track record on delivering these types of materials. For example, Burton's previous ESRC award focused on theoretical developments in face recognition, but he also published a graphical face-manipulation environment (InterFace) as a side-benefit, which is freely available and now used by researchers worldwide.
4) Engagement of wider public
Psychological research has great intuitive appeal for members of the general public, who are also the ultimate indirect beneficiaries of the knowledge developed in this project, through improvements in airport and border security. VR is also rapidly gaining popularity in non-academic contexts, such as computer gaming. To engage these beneficiaries, we will present our research at public science events, such as the British Science Festival, British Science Week, and 'Pint of Science' festivals. In addition, news from this project will be disseminated via UoK and UoY's Psychology Twitter accounts.
Organisations
Publications
Fysh MC
(2022)
Avatars with faces of real people: A construction method for scientific experiments in virtual reality.
in Behavior research methods
Fysh MC
(2023)
Understanding face matching.
in Quarterly journal of experimental psychology (2006)
Fysh M
(2023)
EXPRESS: Queues, Crowds, and Angry Mobs: Face Identification Under Distraction in a Virtual Airport
in Quarterly Journal of Experimental Psychology
Fysh M
(2022)
Molistic processing in facial image comparison
in Applied Cognitive Psychology
Claydon J
(2022)
Facial comparison behaviour of forensic facial examiners
in Applied Cognitive Psychology
Bindemann M
(2022)
Face identification in the laboratory and in virtual worlds.
in Journal of Applied Research in Memory and Cognition
| Description | The overarching aim of this project was to develop virtual reality (VR) technology for the psychological study of person perception and identification. VR has developed rapidly in recent years but the implementation of this technology to study human behaviour in scientific experiments remains challenging. NEW RESEARCH METHODS: A key issue here is that virtual characters must provide an experience that captures the important characteristics of real people - otherwise this technology will provide a limited proxy for the real world. In this project we focused on a key problem in this domain - how it is possible to efficiently produce virtual characters of real people (called avatars) that present veridical visual representations of the persons upon which they are based. We have developed an efficient technique to scan real people in 3D, and to process and transform these scans into photo-realistic animated avatars for use in VR. To validate this method, we scanned over 200 people of diverse ages (4-86 years), both sexes, and a range of ethnicities. We then conducted experiments to explore the psychological characteristics of these avatars. For example, we have shown that avatar faces of familiar people are recognised with high accuracy and produce a similarity-space that corresponds closely with real faces. These studies open up the way to conduct psychological experiments on visual perception and social cognition under increased realism in VR. NEW RESEARCH RESOURCES: We then developed a work flow to allow other researchers to adopt these methods. This is detailed in a manual and accompanied with instruction videos and practice materials. All of this work has been published as an Open Access article in Behaviour Research Methods (see Outputs). NEW KNOWLEDGE GENERATED: We then employed these avatars to examine the correspondence of behaviour in complex settings with controlled laboratory experiments. To do so, we used the example of airport security, by comparing person identification at passport control in a virtual reality airport with established laboratory tests of face identity matching. This work shows correspondence between behaviour in laboratory tasks and VR. However, we also found that behaviour in VR deviates in important ways, by introducing biases in observers that influence identification decisions and which mimic behaviour under real-world conditions. These findings highlight the importance of understanding human behaviour under more complex conditions, and show that VR can facilitate the study of this by providing complex but controlled environments for experimentation. The avatar and airport groundwork was extensive and gave us a strong foundation for further experiments. We capitalised on this for the remainder of the project to pursue several lines of inquiry, which are at various stages of the publication cycle. This includes research on crowd behaviour and the frequency of security targets at airports, as well as research into avatar realism, such as whether observers can tell virtual characters from real people and make similar personality judgements to these. |
| Exploitation Route | Researchers from a broad range of disciplines study 'person perception', aiming to understand how we perceive others. Many psychologists study the perceptual aspects of this problem - often focussing on vision, for example in the field of face recognition. Two methodological approaches dominate this research. Laboratory experiments rely on carefully selected visual material, usually presented in isolation without extraneous background or social context. While this approach provides some fundamental insights into person perception, and retains control over 'noisy' environments, some researchers have worried that results may not generalise well into the real-life settings in which we actually interact with people day to day. VR opens up exciting new possibilities for behavioural scientist to conduct experiments in complex and interactive but controlled settings. For such work to be of clear relevance to real-world behaviour, this require avatars that veridically capture the appearance and characteristics of real people. As a result of this ESRC-funded work, we have been able to provide scientists with a validated method for this purpose. While noting that our work was only published in 2022, our avatar construction materials have so far been downloaded by researchers from Australia, Britain, China, France, Germany, Norway, South Korea and Thailand, demonstrating the wide reach of this work. VR technology has much wider potential application than the specific focus of person identification at airports in this research project. For example, there are many applications from medicine to the entertainment industry. Our findings are likely to be of interest to these sectors too. For example, our grant work was paused for a month-long secondment of the research assistant to Target3D, who are world-leaders in motion capture and VR, which was funded by a separate SeNNS/ESRC Industrial Collaboration Fund (see https://www.youtube.com/watch?v=qmk9ByRrNRs). |
| Sectors | Creative Economy,Digital/Communication/Information Technologies (including Software) |