Meeting the challenge of simultaneous talk for cochlear implant users

Lead Research Organisation: University of Sheffield
Department Name: Human Communication Sciences


Although people do not usually talk at the same time, simultaneous speech by two or more speakers is surprisingly frequent. In the typical conversations recorded for our recent AHRC -funded project it occupies 16% of total talking time, 41% of speaker turns being overlapped by another speaker. Simultaneous or overlapping talk is known to be a particular problem for individuals who have a hearing loss, even when using a conventional hearing aid or cochlear implant. Until recently, even in one-to-one settings many users would need optimum conditions in order to hold a satisfactory conversation, e.g. a quiet environment and the communication awareness of both participants that they should avoid talking at the same time. Professionals have steered clear of advising cochlear implant users about how to deal with situations of overlapping talk, on the basis that such a situation would be just too hard to handle. However, recent improvements in the signal processing strategies used in cochlear implants mean that it is now more realistic for users to attempt to engage in conversations where overlapping talk occurs. The aim of this follow-on project is to engage with a group of adult users of cochlear implants in order to develop useful training materials for handling overlapping talk in conversation. These materials will draw mainly on the outputs from our earlier project on overlapping talk, where we have developed a unique corpus and some key findings about overlapping talk in normal conversation. To the best of our knowledge, these will be the first materials that specifically address the problems raised by overlapping talk.

The main objectives of this project are:

1. To identify the specific issues that overlapping talk raises for cochlear implant users
This will be accomplished by direct questioning, via focus group and questionnaire survey; observation of recorded naturalistic conversations; and by exploring linguistic and cultural differences.

2. To develop ways of improving the experience of cochlear implant users
This will involve devising training software and activities for cochlear implant users, in close collaboration with a group of cochlear implant users. The idea is that the implant user will be able to work with the materials on their own, and with their family members, at home. The software will make use of real examples from our collection of recorded conversations, where people are quite often talking in overlap. The software will focus on both listening and speaking. On the listening side, for example, it will allow users to simultaneously hear and visualise the flow of a conversation over time, by presenting speaker activity on a graphical timeline. On the speaking side, it will provide learning tasks that allow users to practise producing cues in their own speech.

3. To promote and disseminate the training software through a special event, a dedicated website and through existing channels for cochlear implant users and professionals.

This project will engage with a small number of cochlear implant users initially, in order to develop software materials that can assist cochlear implant users when dealing with overlapping talk. The project is embedded in the local NHS cochlear implant service, in which two project team members are employed. Initially the outputs of the project will benefit users of that service. They will be disseminated more widely in the UK through the final dissemination event, and through participation in established national meetings for professionals and for cochlear implant users. Other team members will disseminate results at national and international conferences, which will raise awareness of the work within relevant academic communities. The project website will provide global access to the software materials: the project therefore has the potential to enhance the social participation of more than a quarter of a million cochlear implant users worldwide.

Planned Impact

This project is embedded in a local NHS service for cochlear implant users. Its more immediate benefits will be for the users of that service, and also for the professionals who work with them.
The Sheffield Cochlear implant service provides lifelong support to over 60 adult implant users. As technology improves, expectations of performance with implants are higher and in addition users wish to participate at a higher level in everyday communication. As part of the service's ongoing audit of performance of implants, more complex aspects of communication skills have been identified, which could benefit from further rehabilitation, improving quality of life, and occupational skills. In addition new technology is opening up options for access to rehabilitation tools at home and allowing more tailored remote access to rehabilitation. However, there is little time or funding, within the remit of an NHS service, to address these issues. This collaboration between the service and the University of Sheffield will provide further methods of supporting patients that can also be facilitated by patients themselves.

This Follow-on project for impact and engagement will initially engage with a small number of CI users in Sheffield , in order to develop software materials for CI users who, like other hearing aid users, experience problems when trying to deal with simultaneous or overlapping talk. Until recently, even in one to one settings most users would need optimum conditions in order to have a satisfactory conversation, e.g. a quiet environment and communication awareness of both participants not to overlap in conversation. However, because of recent technical improvements in CIs, many users are now able to participate in more complex communication settings. Thus the envisaged proposal is now highly relevant to improving everyday performance with implants. This will enhance the social participation of CI users.The Cochlear Implant team will facilitate access to users, provide expertise regarding implant technology and ensure any work is patient focused in both its ease of use and delivery, by involving users in this process at all stages.

The Sheffield CI service is partnered with 'Yorkshire Cochlear Implant Service' centred in Bradford and has close relationships with other CI services nationally,therefore training materials can be disseminated to a large group of users via these clinical links. Outputs will be disseminated more widely in the UK through the end-of project engagement event, and through the participation of team members in national and international meetings. The project website will provide global access to the software materials: worldwide there are at least a quarter of million people using CIs.

It is envisaged that the direct benefit to users of the software will be increased social participation, as a result of greater access to conversations.There may also be a benefit to service providers, in terms of improved service delivery:where a user can train him or herself on conversational skills at home using a computer, this potentially reduces the pressure on staff time.
Description This grant was awarded not for research but for impact and engagement.The objectives of this project are listed, followed by progress report on each one.

1) To identify the specific issues that simultaneous (overlapping) talk raises for Cochlear Implant (CI) users. This will be accomplished by the following means:

a) web-based questionnaire survey of CI users;

This has been developed through iterative consultation with our user group,.

b) direct questioning of a focus group of CI users;

Eight focus groups sessions were held with five CI users. From these we have learnt a great deal about the issues they encounter, as well as individual differences.

c) observation of recorded naturalistic conversations involving CI users and comparison with existing corpus of overlaps in recordings of hearing participants;

This can only be done informally under the terms of the current grant and permissions. We have nevertheless learnt a lot already from our observation and recording of the Focus group sessions.

d) identifying how linguistic and cultural differences may impinge on the experience of overlapping talk by CI users, by drawing on findings of previous AHRC project.

This has not been specifically taken forward , as our five users are relatively homogeneous in terms of cultural background.

Findings related to the above aims were fed back a the end-of-award engagement event both by the project team and by the five CI users. This created a good deal of interest among the audience of health professionals and researchers.

2) To devise training software for skill development of CI users in conversational settings:

a) in LISTENING to overlapping talk, based on examples of real overlaps collected in the previous AHRC-funded project;

We developed several listening tasks and trialled them with our CI users. We selected the most suitable ones and incorporated them into the Talking in Time software.

b) in SPEAKING in situations of overlapping talk, drawing on the findings from the AHRC project about how speakers adjust their speech delivery in the context of overlap;

We developed speaking tasks and trialled them with our CI users. We have incorporated some of these into the Talking n Time software. Some aspects of the speaking task software are still under development and will be completed mid 2016.

c) by involving a focus group of CI users iteratively to evaluate the software and advise on its development.

See 2a and 2b above

3) To promote and disseminate the training software:
a) by working with the focus group members, their families and the professionals who support them;

We worked with the CI users themselves,

c) through a special event for local CI users, families and professionals at the end of the project;
This took place in February 2015. and was attended by family members and professionals.

d) through presentations at established fora of CI users and professionals who support them;
We have made several presentations at relevant fora for professionals.

e) through presentations at relevant academic conferences.
We have made several presentations at relevant academic meetings.
Exploitation Route The training materials will provide an inexpensive yet effective way to improve the experience of CI users worldwide, thus increasing their social participation and quality of life. The proposed project is embedded in the local NHS CI service, which employs two project members. Initially, we have engaged with the users of that service, and they will be the first beneficiaries of the project's outputs. the project outputs were disseminated more widely in the UK through the final engagement event, which had regional participation, and through participation in national meetings (e.g., annual conferences of the British Society of Audiology and British Cochlear Implant Group). The project website will provide global access to the software materials. We therefore anticipate that our materials will be used widely; and we hope that our approach may be adopted by other clinical research groups in different linguistic and cultural contexts. Another very large potential user group is that of (normal hearing) advanced learners of English as a Second / Foreign Language.
Sectors Education,Healthcare

Description Emeritus fellowship
Amount £22,700 (GBP)
Funding ID EM 2015 017 
Organisation The Leverhulme Trust 
Sector Charity/Non Profit
Country United Kingdom
Start 02/2016 
End 01/2018
Title A Corpus of Spontaneous Multi-Party Conversation in Bosnian Serbo-Croatian and British English. 
Description We have recorded an audio/visual corpus of conversations held in two European languages - Standard Southern British English and Bosnian Serbo-Croatian, totaling about 4 hours per language. There were four young adult participants for each language. The participants, who already knew each other, were recorded in a naturalistic setting in their home country. Each speaker was recorded on a separate audio channel, so that their talk can be analysed acoustically even when speaking in overlap. For a full description of the corpus, see Kurtic et al. (2012). All the recordings have been transcribed orthographically in ELAN and each instance of overlap has been annotated on a number of parameters. In due course we will make an annotated database of our audio/visual recordings available to other researchers via the Internet, and provide the facility for researchers to share their own annotations of the corpus. In this way, the corpus of recordings will be an evolving resource that will continue to benefit the research community beyond the life of the project. 
Type Of Material Database/Collection of data 
Year Produced 2013 
Provided To Others? Yes  
Impact In aour current AHRC follow-onproject we are developing software based training materials that promote key conversational competencies in cochlear implant users. We are designing graded tasks to enable users to repeatedly practise (i) crucial listening skills and (ii) speaking skills fundamental to multi-party conversation . These materials draw directly on the outputs from our earlier AHRC-funded project on overlapping talk, where we have developed a unique corpus and some key findings about overlapping talk in conversation. To the best of our knowledge, this will be the first training software for hearing implaired users that specifically addresses the problems raised by overlapping talk. Although inthis project we are working with English CI users, we have made use of our Bosnian corpus with them, to illustrate universal conversational practices. 
Description Sheffield Cochlear implant service 
Organisation Sheffield Teaching Hospitals NHS Foundation Trust
Country United Kingdom 
Sector Public 
PI Contribution Our AHRC funded follow on project for impact and engagement is a collaboration between Sheffield Teaching Hospitals NHS Trust and the University of Sheffield, to develop conversation training software for cochlear implant users. The university team provides linguistic, phonetic and software development expertise.
Collaborator Contribution The NHS team provides expertise in audiology and speech and language therapy . They have recruited five CI users for an ongoing series of focus groups where the training materials are tried out. They run the focus group sessions. They also are implementing the dissemination of a web based questionnaire to CI users about their experiences of conversation.
Impact Disciplines: audiology; speech and language therapy; computer science; phonetics; linguistics Outputs; see publications on researchfish associated with this AHRC award
Start Year 2014
Title 'Talking in Time' 
Description Talking in Time: a self-administered training programme for individuals wishing to develop skills in conversational turn-taking. The software was originally devised for and in collaboration with adult users of cochlear implants. Aim: To enhance the user's potential for participating in conversations (a) passively as recipient / listener; (b) actively as speaker. Objectives: 1) To raise user's awareness of how conversation works, with specific reference to turn-taking and overlap 2) To provide structured practice in listening and speaking in a variety of turn-taking contexts MODULE 1: Taking part in conversations Phase 1: How to use 'Talking in Time' Phase 2: How conversations work: Mechanics of turn-taking Phase 3: Participating in conversations effectively: Cues to turn-taking MODULE 2: Clear turns Phase 1: Awareness - identifying the gender of speakers in multiparty talk Phase 2: Listening - identifying a clear (non-overlapping) turn taken on time vs. turn taken late Phase 3: Speaking - taking a clear turn on time (i.e. not early or late) MODULE 3: Competitive overlaps Phase 1: Awareness - identifying multiple speakers talking in sequence vs. one speaker Phase 2: Listening - distinguishing competitive overlaps from clear transitions Phase 3: Speaking - taking a competitive overlapping turn MODULE 4: Accidental overlaps Phase 1: Awareness - becoming familiar with the features that characterise competitive, collaborative and accidental overlaps Phase 2: Listening - identifying if an overlap is intentional-competitive vs. intentional-collaborative vs. accidental Phase 3: Speaking - dropping out immediately on finding oneself accidentally in overlap 
Type Of Technology Software 
Year Produced 2016 
Open Source License? Yes  
Impact A preliminary version of the software was demonstrated at the end-of-project event, where it was trialled by a small group of cochlear implant users. 
Description One-day engagement event (Sheffield) 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Professional Practitioners
Results and Impact The one-day workshop provided an introduction to the Talking in Time training software, developed in our AHRC-funded follow-on project for impact and engagement, in collaboration with cochlear implant users from the South Yorkshire region. The group of cochlear implant users who participated in the project focus groups also attended this workshop. Their participation in question and answer sessions was probably the highlight for many audience members, due to the insights they gave into how it is for them to converse with hearing people every day. Workshop participants had the opportunity to see a demonstration of the software and to use it themselves at the end of the day. Audience feedback was very positive.
Year(s) Of Engagement Activity 2015
Description iCog Workshop: Turn-taking in conversation: a multi-disciplinary approach 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? Yes
Geographic Reach Local
Primary Audience Postgraduate students
Results and Impact Following on from two keynote talks from different displinary prerspectives , Emina introduced the Sheffield Free Talk conversational dataset and suggested points for discussion. The workshop participants then gathered in small groups and spent an hour uncovering varying approaches to segmentation, turn-taking cues, and potential applications of the two datasets in their particular lines of research.

We received some very encouraging feedback from participants at the workshop, and are pleased that the day seemed to be received so well. We are not planning a formal follow-up to the workshop itself, though we will have an end-of-project event in February 2015 for when our current AHRC grant expires. Workshop participants are welcome to return at that stage to find out more about what progress we have made.
Year(s) Of Engagement Activity 2014