Gesture Recognition in Aphasia Therapy

Lead Research Organisation: City University London
Department Name: Centre for HCI Design

Abstract

Aphasia is a language disorder, usually caused by stroke. It affects a substantial number of people; there are about 250,000 people living with aphasia in the UK and approximately 45,000 new cases each year. People with aphasia have difficulty with all aspects of communication: speaking, reading and writing. The negative effects of aphasia can be immense and are not confined to the affected person but are also felt by the immediate family and social circle.One approach advocated by therapists is for the aphasic person to communicate using gestures, but this also poses challenges as the ability to produce and understand gestures may be impaired. In recent research we have investigated whether gesture production can be enhanced by therapy and the initial evidence from face-to-face therapy is encouraging. However gesture therapy is resource-intensive and this has motivated our proposal.Our goal is to develop and pilot an innovative gesture training tool to be used in aphasia therapy at home. This will build on our existing work on non-computer based therapy and there is no previous published research in this area. Our approach to achieving this goal will be to refine and adapt existing gesture recognition technology, incorporating it within a training package that we will develop. In so doing, we will refine and exploit existing research into gesture recognition and gesture-based interaction, utilising mainstream interaction devices such as the Nintendo Wii remote and open source software. We will investigate appropriate forms of gesture recognition and feedback for this user community.The development of the tool will address several fundamental issues: it needs to recognise a substantial set of gestures that are relevant to the aphasic user and can be trained by the therapist; it must not be over-sensitive as gesture production is likely to be variable and inaccurate; we need to understand and incorporate appropriate forms of training and feedback in the tool.Research in the wild is integral to this proposal. Close engagement with the user community will be facilitated by our project partner, the Stroke Association. Aphasic people will be employed as consultants in exploring and evaluating gesture recognition technologies as the design and development of the therapy tool progresses. In the later stages of the project, a pilot study will be conducted with 10 people who have severe aphasia to investigate the therapeutic efficacy of the tool as well as its broader acceptability.The potential transformational impact of using gesture-recognition technology in therapy is significant. Given the numbers of people affected by aphasia, the provision of a therapeutically effective, low-cost, gesture training tool would be of substantial benefit.

Planned Impact

Who will benefit from this research? The main beneficiaries will be people with aphasia. Aphasia affects about 250,000 people in the UK and profoundly impairs quality of life. Secondary beneficiaries will be caregivers of people with aphasia, a group at risk of negative health outcomes [1], and rehabilitation staff and services, particularly speech and language therapists. Other disabled groups, e.g. people with intellectual disorders, may benefit from similar creative uses of interaction technologies. HCI and LCS researchers will benefit from exposure of ideas to the user community. IT companies may benefit from commercial development and exploitation of the tool. How will they benefit? Aphasic people will benefit by acquiring a new therapeutic tool that will help them learn a vocabulary of gestures and hence communicate with family members and friends. Wii applications with other disabled groups suggest that there may be further benefits for mood and wellbeing [2]. Family members will also benefit from the communicative gains. We will select gestures that relate to caregiving (e.g. 'drink' and 'toilet') to help alleviate carers' burden. Rehabilitation staff will acquire a new therapeutic tool to use with their aphasic clients, based on cheap and widely available technology. As the tool is self administered it should be highly cost effective. This is important, given the scarcity of face-to-face therapy: many clinical settings offer only minimal provision [3]. Timescales: The gesture training tool will be available 12 months into the project, when it will be piloted with 10 aphasic people. If the results are positive, we will make the tool widely available to the stroke and clinical communities. Thus there will be immediate benefits. We will also seek funding for a large follow up trial with at least 50 users. This will provide class 1 evidence about the efficacy of the tool. The current and follow up study will be informative for rehabilitation staff working with other client groups, and may promote further adaptations. Such subsidiary benefits are long term (3-8 years). What will be done to ensure that they have the opportunity to benefit? Engagement with aphasic people is integral to this research. Aphasic consultants will advise on the development of the technology, and different pilot participants will test its learning potential and acceptability. We will consult and inform participants' main carers. Results will be disseminated through the 30 London stroke groups affiliated to the Stroke Association and a travelling road show will give group members taster opportunities with the technology. Results of the study will be published in outlets designed for people with aphasia, such as 'Aphasia Now' (website). Additional activities will engage clinical staff. We will stage a CPD training event, to report on the project findings and give clinicians access to the technology. Practising therapists will be invited to trial the therapy tool with their clients, and give feedback. We will target both clinical and academic audiences when writing for publication and making conference presentations. Any Intellectual Property arising from the project will be protected through patent applications by the University's dedicated Research and Enterprise (CREU) team. We anticipate further development of the tool following feedback from the pilot and will work with CREU to achieve this through technology transfer opportunities. The project team is very experienced in engaging user communities and carrying out high impact research, e.g. exploring computer applications in healthcare (HCID) and outcomes from novel therapies (LCS). Thus the team offers an ideal synergy of skills for this project. 1. Bakas, T. et al (2006) Rehabilitation Nursing, 31 (1) 33 - 42 2. Science Daily (2009) Nintendo Wii may enhance Parkinson's Treatment. 3. Enderby, P. et al (2002) Clinical Rehabilitation, 16, 604-608.
 
Description Aphasia is a communication disorder that sometimes occurs after a stroke and can affect all aspects of a person's use of language: speaking, reading, writing and comprehension. One possibility for people with aphasia is to use non-verbal communication strategies such as gestures instead but their ability to produce gestures may also be impaired. Therapy to enhance the production of gestures is usually delivered by speech and language therapists; the GReAT project aimed to provide a novel, computer-based alternative.

The overall objective of GReAT was to investigate, prototype and trial a gesture therapy tool for use at home by people with aphasia. The key findings from the research lay in three areas:

1. Interaction design for people with aphasia
Current guidelines for designing accessible interactive technologies provide little advice on designing for aphasia; the first of our key findings was a contribution to this area. We ran a series of participatory design workshops with 5 people who have aphasia who were employed as consultants on the project. The workshops explored how people with aphasia interact with technologies, where they are successful and where they face challenges. The results were reported as a set of initial design principles (Galliers et al, 2011) and as techniques and lessons learned for facilitating participation of people with aphasia in design (Galliers et al, 2012).

2. GeST: a gesture therapy tool for people with aphasia
We designed and implemented GeST, a prototype gesture therapy tool to help people with aphasia improve their gesturing (Marshall et al, 2013) (Wilson et al, 2015). GeST works by demonstrating communicative gestures (such as phone, umbrella, glasses) to the user, detecting the user's attempts to produce the gestures and providing motivational feedback. It incorporates computer vision-based gesture recognition technology and a 3D world to provide an engaging interactive experience. GeST is novel both as a mechanism for providing gesture therapy and in its use of a 3D virtual world in a therapy tool for people with aphasia.

3. Therapy study
An eight-month, home-based study was undertaken to investigate the therapeutic efficacy of GeST (Marshall et al, 2013). The study involved 10 participants with severe aphasia; 9 successfully completed the therapy. The primary aim was to determine whether GeST improved gesture production; secondary aims were to investigate whether GeST improved production of spoken words, its acceptability and its usability.
Results showed that gestures practised with GeST improved significantly after therapy and the gain was maintained. The gains only occurred on items that were practised with GeST and therapist support. There was no generalisation to unpractised gestures and no effect on spoken words. Usability results were positive: results showed that GeST provoked interest, motivation and a sense of fun.
Exploitation Route 1. The principles for accessible design can be applied by practitioners involved in the design and implementation of a broad range of interactive technologies for people with communication impairments. These principles are being developed further in a subsequent research project (EVA).

2. The GeST prototype could be developed into a robust software package that could be made widely available to therapists and people with aphasia. We are exploring options for follow-on funding for this purpose.

3. A PhD student undertook a further, more substantial, empirical study of the GeST tool involving 20 people with aphasia. This work was published in late 2016.

4. The research has, and will continue to, raise awareness of the potential role of digital technology in aphasia therapy. It has demonstrated a novel approach to utilising technology which has attracted significant interest from both academic researchers and practitioners.
Sectors Digital/Communication/Information Technologies (including Software),Healthcare

 
Description The findings have: - Inspired further research into technologies, including therapeutic technologies, for people with aphasia. - Been taken forward into a doctoral research project that used the GeST software in further trials and led to further publications. - Been disseminated directly to practitioners and user groups, contributing to a heightened awareness of the therapeutic potential of technology for people with aphasia. - Contributed to an increasing body of knowledge on how to design technology that is accessible to people with aphasia. E.g. a presentation on UX design for people with language impairments to the UXPA UK in Feb 2016 and workshops on "Language-Light UX" delivered in 2016 and 2017. - Opportunities for making the software available to practitioners are being explored.
First Year Of Impact 2012
Sector Digital/Communication/Information Technologies (including Software),Healthcare
Impact Types Societal

 
Description City University Alumni Fund
Amount £1,000 (GBP)
Organisation City, University of London 
Sector Academic/University
Country United Kingdom
Start 09/2011 
End 12/2012
 
Description City University London Doctoral Studentship
Amount £50,000 (GBP)
Organisation City, University of London 
Sector Academic/University
Country United Kingdom
Start 04/2012 
End 03/2015
 
Description Evaluating the effects of a virtual communication environment for people with aphasia
Amount £204,898 (GBP)
Funding ID TSA2011/10 
Organisation Stroke Association 
Sector Charity/Non Profit
Country United Kingdom
Start 09/2012 
End 08/2015
 
Description Exploring the Accessibility of Second Life for People with Aphasia (EASE)
Amount £5,000 (GBP)
Organisation City, University of London 
Department Research Pump-Priming Fund
Sector Academic/University
Country United Kingdom
Start 04/2012 
End 10/2012
 
Description Research grant. Remote Aphasia Therapy: A Feasibility Study
Amount £53,263 (GBP)
Organisation The Tavistock Trust for Aphasia 
Sector Charity/Non Profit
Country United Kingdom
Start 10/2012 
End 09/2013
 
Description Project Collaborator 
Organisation Stroke Association
Country United Kingdom 
Sector Charity/Non Profit 
PI Contribution Co-design of the GeST therapy tool, implementation of GeST, field study with people who have aphasia, dissemination activities including road shows at stroke groups.
Collaborator Contribution Participated in advisory meetings, provided us with access to stroke groups for recruitment and dissemination, supported dissemination activities.
Impact All project outputs.
Start Year 2010
 
Description GReAT Project Video 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact Video introducing the context within which the research was undertaken and main achievements of the project.

http://vimeo.com/40081415.

International contacts.
Year(s) Of Engagement Activity 2012
URL http://vimeo.com/40081415
 
Description GeST Road Shows 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Participants in your research and patient groups
Results and Impact A series of events where GeST was demonstrated at Stroke Clubs. Resulted in discussion and the recruitment of research participants.

New research participants and contacts with Stroke Group co-ordinators.
Year(s) Of Engagement Activity 2012
 
Description Gesture-Based Interaction In The Rehabilitation of Stroke Survivors 
Form Of Engagement Activity Participation in an open day or visit at my research institution
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Professional Practitioners
Results and Impact Research presentation.

n/a
Year(s) Of Engagement Activity 2011
 
Description Invited Research Presentation at UK Stroke Forum 2013 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Shared information. Sparked interest and discussion.

Raised profile and interest in our work.
Year(s) Of Engagement Activity 2013
 
Description Presentation at Centre for HCI Design Open Day 2012 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Professional Practitioners
Results and Impact Talk sparked discussion.

New contacts.
Year(s) Of Engagement Activity 2012
 
Description Press release 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Media (as a channel to the public)
Results and Impact Press release.

n/a
Year(s) Of Engagement Activity 2011
 
Description Research Presentation at 15th International Aphasia Rehabilitation Conference. 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other academic audiences (collaborators, peers etc.)
Results and Impact Discussion and dissemination.

Increased international profile.
Year(s) Of Engagement Activity 2012
 
Description Research Presentation at British Aphasiology Society Therapy Symposium, Sept 2012 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Other academic audiences (collaborators, peers etc.)
Results and Impact Questions and discussion.

n/a
Year(s) Of Engagement Activity 2012
 
Description Research Presentation at RAatE 2012 Conference 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Other academic audiences (collaborators, peers etc.)
Results and Impact Dissemination and contacts.

Raised profile.
Year(s) Of Engagement Activity 2012
 
Description Research Presentation to Frenchay Computers in Therapy SIG. GeST: Computer Delivered Gesture Therapy for People with Aphasia 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Professional Practitioners
Results and Impact Research and practice presentation.
Presentation to Frenchay Computers in Therapy SIG.
Talk sparked questions and discussion.

Talk sparked questions and discussion.
Year(s) Of Engagement Activity 2012
 
Description Research Seminar: Engaging and Accessible Interaction Design for People with Aphasia 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Other academic audiences (collaborators, peers etc.)
Results and Impact Research seminar at Brunel University.

n/a
Year(s) Of Engagement Activity 2012
 
Description Research Seminar: Fun and Consistency: Engaging Interaction Design for People with Aphasia 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Other academic audiences (collaborators, peers etc.)
Results and Impact Research seminar at UCL, November 2011.

n/a
Year(s) Of Engagement Activity 2011
 
Description Video of GeST, a software tool for gesture therapy 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact http://vimeo.com/40089296.

International contacts.
Year(s) Of Engagement Activity 2012
URL http://vimeo.com/40089296
 
Description Video: Demonstration of the GeST tool 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Video demonstrating the first prototype of GeST.

Contacts.
Year(s) Of Engagement Activity 2011
URL http://vimeo.com/37727969