Ultrax2020: Ultrasound Technology for Optimising the Treatment of Speech Disorders.

Lead Research Organisation: University of Edinburgh
Department Name: Centre for Speech Technology Research

Abstract

Speech Sound Disorders (SSDs) are the most common communication impairment in childhood; 16.5% of eight year olds have SSDs ranging from problems with only one of two speech sounds to speech that even family members struggle to understand. SSDs can occur in isolation or be part of disability such as Down syndrome, autism or cleft palate. In 2015, the James Lind Alliance identified improving communication skills and investigating the direction of interventions as the top two research priorities for children with disabilities. Our programme of research aims to fulfil this need by developing technology which will aid the assessment, diagnosis and treatment of SSDs.

Currently in Speech and Language Therapy, technological support is sparse. Through our previous work in the Ultrax project we showed that by using ultrasound to image the tongue in real-time, children can rapidly learn to produce speech sounds which have previously seemed impossible for them. Through this project, we developed technology that enhances the ultrasound image of the tongue, making it clearer and easier to interpret. Ultrax2020 aims to take this work forward, by further developing the ultrasound tongue tracker into a tool for diagnosing specific types of SSDs and evaluating how easy it is to use ultrasound in NHS clinics. The ultimate goal of our research is that Ultrax2020 will be used by Speech and Language Therapists (SLTs) to assess and diagnose SSDs automatically, leading to quicker, more targeted intervention.

Normally speech assessment involves listening to the child and writing down what they say. This approach can miss important subtleties in the way children speak. For example, a child may try to say "key" and it may be heard as "tea". This leads the SLT to believe the child cannot tell the difference between t and k and select a therapy designed to tackle this. However, ultrasound allows us to view and measure the tongue, revealing that in many cases children are producing imperceptible errors. In the above example, an ultrasound scanner placed under the chin shows that the child produces both t and k simultaneously. Identification of these errors means that the SLT must choose a different therapy approach. However, ultrasound analysis is a time consuming task which can only be carried out by a speech scientist with specialist training. It is a key output of Ultrax2020 to develop a method for analysing ultrasound automatically, therefore creating a speech assessment tool which is both more objective and quicker to use.

Building on the work of the Ultrax project, where we developed a method of tracking ultrasound images of the tongue, Ultrax2020 aims to develop a method of classifying tongue shapes to form the basis of an automatic assessment and a way of measuring progress objectively. We are fortunate to already have a large database of ultrasound images of tongue movements from adults and primary school children, including those with speech disorders, on which to base the model of tongue shape classification and to test its performance. At the same time, we will evaluate the technology we develop as part of Ultrax2020 by partnering with NHS SLTs to collect a very large database of ultrasound from children with a wide variety of SSDs. In three different NHS clinics, SLTs will record ultrasound from over 100 children before and after ultrasound-based speech therapy. This data will be sent to a university speech scientist for analysis and feedback to clinicians recommending intervention approaches. Towards the end of the project, we will be able to compare this gold-standard hand-labelled analysis with the automatic classification developed during the project. At the conclusion of our research project we will have developed and validated a new ultrasound assessment and therapy tool (Ultrax2020) for Speech and Language Therapists to use in the diagnosis and treatment of SSDs.

Planned Impact

Communication is a fundamental human right - poor communication skills cost individuals and society. Speech Sound Disorders (SSD) are the most common communication impairment: 16.5% of eight year olds have SSDs ranging from problems saying just one or two speech sounds to speech that even close family members struggle to understand. Technology to help children and adults with SSDs is limited. However, in the previous Ultrax project we showed that by using ultrasound to image the movements of the tongue in real-time, children can rapidly learn to produce speech sounds which have previously seemed impossible for them. Through that project, we developed technology that enhances the ultrasound image of the tongue, making it clearer and easier to interpret. Ultrax2020 aims to take this work forward, by further developing the ultrasound tongue tracker into a tool for diagnosing specific types of SSDs and evaluating how easy it is to use ultrasound in NHS clinics. The ultimate goal of our research is that Ultrax2020 will be used by Speech and Language Therapists (SLTs) to assess and diagnose SSDs automatically, leading to quicker, more targeted intervention. There are three large groups set to benefit from our research:

1. Children and Adults with Speech Sound Disorders
2. Speech and Language Therapists
3. Students of phonetics or Speech and Language Therapy

At the outset of the Ultrax project in 2011 there was very little work showing whether ultrasound-based speech therapy might be an effective treatment and no work using ultrasound to diagnose specific types of SSDs. Through the Ultrax project, and a related clinical project, we have shown that ultrasound can be used to help children with SSDs which were previously thought to be untreatable. The impact on children's quality of life is considerable, and feedback from parents has been positive and enthusiastic. In the Ultrax2020 project, over 100 children with SSDs will be treated by NHS SLTs, bringing direct benefit to those individuals in terms of improved quality of life. More importantly, by evaluating ultrasound in clinics, as opposed to the university lab, we can begin to prove that ultrasound is a practical treatment. Ultimately, a way of automatically assessing and diagnosing SSDs will save SLTs considerable time. During the project, the NHS SLTs involved will benefit from ultrasound analysis performed by a speech scientist for every client they assess. They will be able to use that analysis to plan treatment that is more likely to bring about positive gains in children's speech.

At the outset of the project we will run a training day for SLTs on using ultrasound for speech therapy. We have experience of training others to use ultrasound and we will aim to train SLTs in such a way that they can train other SLTs in their own teams. Since three NHS clinics across Scotland will be using our ultrasound technology, we will organise feedback sessions so that SLTs can advise us on the best way to make software suitable for them to use. It is a major part of this project to collect a database of ultrasound tongue images from children with a wide variety of SSDs. This will form the largest database of its kind in the world and will be an invaluable resource for other researchers who want to know more about the nature of speech disorders. We will be able to use this database ourselves for training student SLTs or for student research projects.

Ultrax2020 has strong commercialisation prospects. Articulate Instruments Ltd is one of only a handful of companies worldwide that supplies technology such as this to the speech therapy market. They have been close collaborators since the beginning of the previous Ultrax project and are committed to taking Ultrax2020 forward, having already developed an app specifically for ultrasound-based speech therapy; they are keen to develop this further for assessment, diagnosis and treatment of SSDs.

Publications

10 25 50

publication icon
Cleland J (2020) The Impact of Real-Time Articulatory Information on Phonetic Transcription: Ultrasound-Aided Transcription in Cleft Lip and Palate Speech. in Folia phoniatrica et logopaedica : official organ of the International Association of Logopedics and Phoniatrics (IALP)

publication icon
Sugden E (2019) Systematic review of ultrasound visual biofeedback in intervention for speech sound disorders. in International journal of language & communication disorders

publication icon
Zhang J (2021) TaLNet: Voice Reconstruction from Tongue and Lip Articulation with Transfer Learning from Text-to-Speech Synthesis in Proceedings of the AAAI Conference on Artificial Intelligence

 
Description Ultrax2020 aims to develop a method of classifying tongue shapes to form the basis of an automatic assessment and a way of measuring speech therapy progress objectively. We are fortunate to already have a large database of ultrasound images of tongue movements from adults and primary school children, including those with speech disorders, on which to base the model of tongue shape classification and to test its performance, which has been released as Ultrasuite. At the same time, we will evaluate the technology we develop as part of Ultrax2020 by partnering with NHS SLTs to collect a very large database of ultrasound from children with a wide variety of SSDs. In four different NHS clinics, SLTs will record ultrasound from over 100 children before and after ultrasound-based speech therapy. This data will be sent to a university speech scientist for analysis and feedback to clinicians recommending intervention approaches. Towards the end of the project, we will be able to compare this gold-standard hand-labelled analysis with the automatic classification developed during the project. At the conclusion of our research project we will have developed and validated a new ultrasound assessment and therapy tool (Ultrax2020) for Speech and Language Therapists to use in the diagnosis and treatment of SSDs.

Our findings so far include:

1. Development of a software prototype for SLT use. Through an iterative cycle of technology integration and testing with SLT partners, we are preparing a prototype clinical software package.

2. Collection of ultrasound tongue imaging data in an SLT context.
New data is required to complement the ultrasound data we have now, reflecting the new diagnostic aim of this project. The SLT partners are collecting this, and it is being annotated, analysed, and used for machine learning and signal processing.

3. Formulation of an assessment and data collection protocol.
To support the SLT data collection work, we have devised a protocol for assessment and data collection. This includes assessment wordlists, shape templates, and a therapy manual. In addition, we have held training and feedback days for the Ultrax2020 ultrasound visual biofeedback methodology.

4) Development of automatic identification of speech segments of interest.
We have developed signal processing and machine learning algorithms to identify segments of interest from ultrasound and speech recordings.

5) Provision of advice and training for ultrasound-based speech therapy.
We are giving advice and training on clinical use of ultrasound to our clinical partners, and to the wider SLT community as a general policy of advocacy.

6) Provision of data as a resource to the research community.
We have released the Ultrasuite repository of ultrasound and acoustic data from child speech therapy sessions. This also includes a set of annotations, some manual and some automatically produced, and tools to process, transform and visualise the data.

7) Development of new signal processing and machine learning algorithms for the processing of ultrasound and acoustic data from child speech therapy sessions.
This includes algorithms for speaker segmentation, for speech recognition, for probe geometry identification, and for automatic alignment of ultrasound and audio data.
Exploitation Route We are working with a commercial partner (Articulate Instruments Ltd) and four NHS Trusts with the aim of enabling thise findings to be used for children's speech and language therapy.
Sectors Healthcare

URL https://www.ultrax-speech.org/projects/ultrax2020/
 
Description The ultimate goal of our research is that Ultrax2020 will be used by Speech and Language Therapists (SLTs) to assess and diagnose SSDs automatically, leading to quicker, more targeted intervention. There are three large groups set to benefit from our research: 1. Children and Adults with Speech Sound Disorders 2. Speech and Language Therapists 3. Students of phonetics or Speech and Language Therapy Our approach to impact in this project is through working with four NHS trusts and with an SME, Articulate Instruments. Our work with the NHS trusts is focused on requirements capture, data collection, and in the use of the ultrasound technology in therapy. Our primary interactions with the NHS trusts have come through the ongoing collaboration, particularly with one of the project researchers, as well as through annual workshops bringing together project staff and project partners. These have enabled us to understand better the requirements of clinical partners, and have informed the design of our research. We work very closely with Articulate Instruments at a technical level, with collaborations on implementing new algorithms in their commercial product. This is the main route through which research advances made in the project can reach users.
First Year Of Impact 2017
Sector Healthcare
Impact Types Societal,Economic

 
Description Consultancy: Training practitioners in ultrasound biofeedback therapy
Amount £850 (GBP)
Organisation Morehouse College 
Sector Academic/University
Country United States
Start 01/2020 
End 01/2020
 
Description Silent Speech Interfaces for all - recognising speech from ultrasound images of the tongue (Research Incentive Grant)
Amount £14,927 (GBP)
Funding ID RIG008585 
Organisation Carnegie Trust 
Sector Charity/Non Profit
Country United Kingdom
Start 08/2019 
End 05/2020
 
Description SonoSpeech Cleft Pilot: A Pilot Randomised Control Trial of Ultrasound Visual Biofeedback Versus Standard Intervention for Children with Cleft Lip and Palate
Amount £217,471 (GBP)
Funding ID TCS/20/02 
Organisation Chief Scientist Office 
Sector Public
Country United Kingdom
Start 06/2021 
End 05/2023
 
Description Speech Therapy Animation and imaging Resource (STAR)
Amount £242,037 (GBP)
Funding ID ES/V012401/1 
Organisation Economic and Social Research Council 
Sector Public
Country United Kingdom
Start 06/2021 
End 06/2023
 
Title UltraSuite Repository 
Description Ultrasuite is a collection of ultrasound and acoustic speech data from child speech therapy sessions. The current release includes three datasets, one from typically developing children and two from speech disordered children. Ultrasuite also includes a set of annotations, some manual and some automatically produced, and tools to process, transform and visualise the data. 
Type Of Material Database/Collection of data 
Year Produced 2018 
Provided To Others? Yes  
Impact N/A 
URL https://ultrasuite.github.io
 
Description Articulate Instruments 
Organisation Articulate Instruments Ltd.
Country United Kingdom 
Sector Private 
PI Contribution Open source release of software and data for ultrasound analysis of speech
Collaborator Contribution Articulate Instruments have a current focus on the use of ultrasound for visualising tongue movements. They have a an application to support tongue tracking and through the Ultrax2020 collaboration they will be able to further develop AI and signal processing components of their application.
Impact 1/ Ultrasound tongue tracking software 2/ Ultrasound/Speech recognition software 3/ Ultrasound/Speech segmentation software 4/ Utility tools for processing ultrasound tongue image data
Start Year 2017
 
Description NHS Glasgow 
Organisation NHS Greater Glasgow and Clyde (NHSGGC)
Department Speech and Language Therapy Service
Country United Kingdom 
Sector Hospitals 
PI Contribution 1/ Development of ultrasound tongue image software for prototype used by speech and language therapy (SLT) partners. Through an iterative cycle of technology integration and testing with SLT partners, we will develop a prototype clinical software package. 2/ To support the SLT data collection work, we have devised a protocol for assessment and data collection of ultrasound tongue image data and associate speech recordings. This includes assessment wordlists, shape templates, and a therapy manual. In addition, we have held training and feedback days for the Ultrax2020 ultrasound visual biofeedback methodology.
Collaborator Contribution 1/ Recording of ultrasound tongue image data in a speech and language therapy (SLT) context. 2/ Evaluation of ultrasound visual biofeedback for SLT in a clinical context
Impact 1/ Development of a protocol for assessment and data collection of ultrasound tongue image data and associate speech recordings 2/ Collection of ultrasound tongue image and associate speech recordings in a clinical context 3/ Evaluation of ultrasound visual biofeedback for SLT in a clinical context
Start Year 2017
 
Description NHS Grampian 
Organisation NHS Grampian
Department Department of Speech and Language Therapy
Country United Kingdom 
Sector Hospitals 
PI Contribution 1/ Development of ultrasound tongue image software for prototype used by speech and language therapy (SLT) partners. Through an iterative cycle of technology integration and testing with SLT partners, we will develop a prototype clinical software package. 2/ To support the SLT data collection work, we have devised a protocol for assessment and data collection of ultrasound tongue image data and associate speech recordings. This includes assessment wordlists, shape templates, and a therapy manual. In addition, we have held training and feedback days for the Ultrax2020 ultrasound visual biofeedback methodology.
Collaborator Contribution 1/ Recording of ultrasound tongue image data in a speech and language therapy (SLT) context. 2/ Evaluation of ultrasound visual biofeedback for SLT in a clinical context
Impact 1/ Development of a protocol for assessment and data collection of ultrasound tongue image data and associate speech recordings 2/ Collection of ultrasound tongue image and associate speech recordings in a clinical context 3/ Evaluation of ultrasound visual biofeedback for SLT in a clinical context
Start Year 2017
 
Description NHS Lothian 
Organisation NHS Lothian
Country United Kingdom 
Sector Public 
PI Contribution 1/ Development of ultrasound tongue image software for prototype used by speech and language therapy (SLT) partners. Through an iterative cycle of technology integration and testing with SLT partners, we will develop a prototype clinical software package. 2/ To support the SLT data collection work, we have devised a protocol for assessment and data collection of ultrasound tongue image data and associate speech recordings. This includes assessment wordlists, shape templates, and a therapy manual. In addition, we have held training and feedback days for the Ultrax2020 ultrasound visual biofeedback methodology.
Collaborator Contribution 1/ Recording of ultrasound tongue image data in a speech and language therapy (SLT) context. 2/ Evaluation of ultrasound visual biofeedback for SLT in a clinical context
Impact 1/ Development of a protocol for assessment and data collection of ultrasound tongue image data and associate speech recordings 2/ Collection of ultrasound tongue image and associate speech recordings in a clinical context 3/ Evaluation of ultrasound visual biofeedback for SLT in a clinical context
Start Year 2017
 
Description UCL Hospitals 
Organisation University College London Hospitals NHS Foundation Trust
Country United Kingdom 
Sector Academic/University 
PI Contribution 1/ Development of ultrasound tongue image software for prototype used by speech and language therapy (SLT) partners. Through an iterative cycle of technology integration and testing with SLT partners, we will develop a prototype clinical software package. 2/ To support the SLT data collection work, we have devised a protocol for assessment and data collection of ultrasound tongue image data and associate speech recordings. This includes assessment wordlists, shape templates, and a therapy manual. In addition, we have held training and feedback days for the Ultrax2020 ultrasound visual biofeedback methodology.
Collaborator Contribution 1/ Recording of ultrasound tongue image data in a speech and language therapy (SLT) context. 2/ Evaluation of ultrasound visual biofeedback for SLT in a clinical context
Impact 1/ Development of a protocol for assessment and data collection of ultrasound tongue image data and associate speech recordings 2/ Collection of ultrasound tongue image and associate speech recordings in a clinical context 3/ Evaluation of ultrasound visual biofeedback for SLT in a clinical context
Start Year 2018
 
Description Invited Lecture 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Undergraduate students
Results and Impact Invited lecture at Manchester University to include elements of social responsibility into the curriculum. Approximately 35 students attended the lecture/workshop regarding the use of ultrasound in speech and language therapy. Attendees provided positive feedback, with many students reporting interest in related subject areas.
Year(s) Of Engagement Activity 2018
 
Description SLT Training Day 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Approx 25 practitioners from Scotland attended a one-day workshop about using ultrasound in the clinical management of speech disorders. Participants reported increased confidence in using the technology, and have since gone on to use the technology in their clinical practice.
Year(s) Of Engagement Activity 2018
 
Description School Engagement Visit 
Form Of Engagement Activity Participation in an open day or visit at my research institution
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Schools
Results and Impact 54 primary school students attended for a school visit to the research organisation to learn about the different types of research and technology being conducted and used at the University. Members of the research team participated in the event by demonstrating ultrasound tongue imaging to smaller groups of students. The students were interested and engaged in the ultrasound technology, and their teachers reported increased interest in science, engineering and technology.
Year(s) Of Engagement Activity 2018
 
Description Ultrasound Biofeedback Workshop 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact 20 speech and language therapists attended a 3-day workshop delivered by members of the research team. The research team delivered training on the use of ultrasound technology in speech and language therapy clinical practice. Several attendees have since reported using ultrasound successfully in their clinical practice.
Year(s) Of Engagement Activity 2019
URL https://ultrax-speech.inf.ed.ac.uk/2019/10/03/ultrasound-workshop-held-at-strathclyde-university/
 
Description Ultrasound Visual Biofeedback Clinician Training Day 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact 20 Speech and Language Therapists from the UK and Ireland attended a three day training course on how to use ultrasound visual biofeedback in their clinical practice. The course was run jointing my SME Articulate Instruments.
Year(s) Of Engagement Activity 2019
 
Description University Visit (US students) 
Form Of Engagement Activity Participation in an open day or visit at my research institution
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Undergraduate students
Results and Impact 30 undergraduate students from US-based universities visited the speech and language therapy department at The University of Strathclyde, and attended a presentation about the ultrasound research we conduct and this project. Attendees were very interested in the research and the potential impact of ultrasound in clinical contexts.
Year(s) Of Engagement Activity 2018