Humans and machines: novel methods for testing speaker recognition performance

As humans, we regularly use the voice as a means of recognising people - for example, when someone calls us on the telephone, or shouts to us from another room. While humans are relatively good at recognising familiar, or at least predictable voices, identifying unfamiliar voices is much more difficult. This is often the task in forensic and investigative contexts. In such cases, a comparison is made of the voices of an unknown criminal and a known suspect, with the ultimate aim of assessing the likelihood that they belong to the same individual. Increasingly, around the world, speaker recognition machines (i.e. pieces of software) are used for these purposes. However, a critical question remains unanswered: do machines recognise speakers in the way that humans do?

This question has received relatively little attention in the literature. The studies that have examined this issue are all small scale and simply compare the results of human recognition with those of machine recognition using overall error rates. However, what is much more important is understanding the contexts in which one method might outperform the other, and whether there is any benefit in combining the approaches. In addressing these issues, our research will provide a better understanding of how speaker recognition machines work and how they might be improved. Further, previous work has overlooked the many factors that may affect human recognition performance, such as cognitive bias. In this project, we assess the variability in human judgements as a function of different amounts of contextual information, especially in the context of a criminal trial where there may be other information pertinent to the case or even a forensic expert providing voice evidence which could influence the decision-making process involved in the speaker recognition task.

In order to compare and combine human and machine responses, we will develop a bespoke computer game that elicits human judgements that are conceptually equivalent to those produced by the machine. In doing so, we will also test the viability of using the voice as the central element in a computer game; an area of computer game development that has received relatively little attention.

The project has a number of specific research questions:
1. How do humans and machines perform at speaker recognition relative to each other, and can we improve performance by combining the two approaches? To what extent, therefore, do these methods capture the same information?
2. In what contexts (using speakers with different regional accents and diverse speech samples with varying durations and recording quality) do humans outperform machines?
3. How do different listener groups perform in speaker comparison tasks? Does familiarity with the regional accent improve performance?
4. To what extent are human judgements affected by contextual information that may occur in a forensic case, such as (i) the knowledge that it is a criminal case, (ii) other evidence from the case, or (iii) a forensic expert's opinion?

The proposed research is strongly applied in nature. The research questions stem from real world issues relating to speaker recognition, particularly in forensic contexts. Thus, our results will be of benefit to a range of non-academic users:

(1) Judicial contexts

The primary beneficiaries of our research will be forensic phoneticians who conduct speaker recognition for the purposes of casework. The results will allow experts to assess the conditions under which automatic systems may perform better or worse, and help experts to explain what such systems are actually doing, in order for courts to make more informed decisions about the evidence. Our results can also be used by experts to make delivery of their expert opinion to courts more accessible in cases involving earwitnesses. We will be able to disseminate our results to practising forensic phoneticians at J P French Associates (the UK's largest and longest established forensic speech and audio laboratory), given their long-standing relationship with our University Department. We will also share our results at the International Association of Forensic Phonetics and Acoustics (IAFPA) conference. IAFPA is the world's only association for forensic phonetics, and the conference attracts a large non-academic audience from private and governmental laboratories, security and law enforcement agencies, and police forces. Since our research will provide insights into the errors made by speaker recognition systems, the results will be of value to commercial developers of speaker recognition systems, principally for helping to improve the performance of such systems in the future. As part of the project, we will discuss our results with Nuance Communications (the largest commercial supplier of speaker recognition technology in the world and the producer of the system we use in this project). Detailed results of the comparison of the effects on decision-making of verbal vs numerical expert witness statements will be directed specifically to beneficiaries in the legal domain. Our results will be discussed with relevant end users in order that the courts are made aware of the potential effect of the wording of expert witness statement on jurors.

(2) Game designers

A central part of this project is to develop a game that has voice as its central component. The voice is an under-utilised modality for computer games and so our project acts as a test of the viability of including voice elements more centrally into games. We will present the findings of our research to the co-director of 'Betajester' (Adam Boyne), a commercial company that develops immersive and interactive digital experiences with a focus on game development. We will also discuss the potential for developing our game for projects in the future.

(3) Teaching and schools

The task of speaker recognition is familiar to the general public through TV shows and films. Its real world applications also make it a fascinating topic for students. We intend to further maximise the impact of this project by engaging with school teachers and students in various ways. We will include a session on the topic of speaker recognition (using findings from the project) as part of a CPD course aimed at A-level English Language teachers. Such courses have run previously at York, and Llamas has been centrally involved in their development. We will also contact schools directly to offer talks about our project for students. Finally, we will offer our findings as a case study for the York English Language Toolkit site; this site provides resources on current research from the Department for teachers and students of A-level English Language.