How was school today...? - Supporting narrative for non-speaking children, a feasibility study.

Lead Research Organisation: University of Dundee
Department Name: School of Computing

Abstract

Our goal is to develop a computer tool which helps children who cannot speak create a story about their day at school. Story telling is an essential aspect of social interaction, and story-telling skills are developed through practice. It is difficult for non-speaking children to get such practice, our tool will help them.More specifically, we want to use various kinds of sensors to acquire information about where the child went, what she did, and who she interacted with; write a computer program which automatically creates a draft story based on this data; and create a story editing and narration interface which lets children edit the draft story and then tell it when they are happy with it. Possible sensors include GPS for tracking where children go, RFID tags for tracking what objects children interact with and hence their activities; and barcode scanners for recording who children interact with. The story-generation software will be based on technology for generating English summaries of data which has been developed in other EPSRC-funded projects such as SumTime, RoadSafe, and BabyTalk. The story editing interface will probably be based on a visual timeline metaphor.Our work will be informed and guided by several user groups, including children, parents, and teachers. We will use a user-centred design philosophy throughout.We will build a simple prototype system at the end of the project, and do a small-scale evaluation with two children. If the feasibility study is successful, we will explore funding opportunities to further develop this concept.
 
Description We have developed a computer tool which helps children who cannot speak create a story about their day at school. How-was-school-today? enables children with disabilities such as cerebral palsy and learning difficulties to have conversations in a faster, more interactive way.

How-was-school-today? uses RFID tags, swipe cards and a voice recording device to gather information on what the child using the system has experienced at school that day. RFID sensors track where the child went during the day; an RFID reader is mounted on the child's wheelchair, and RFID tags are placed around the school, especially in doorways so we can monitor children entering and leaving rooms.

Teachers have RFID swipe cards which they can swipe against a reader, to record that they are interacting with the child; this is more robust than attempting to infer interaction automatically by tracking teachers' position. Teachers can also record interactions with objects (e.g., toys), by using swipe cards associated with these objects. Teachers can also record spoken messages about what happened during the day.


Data analysis combines sensor-derived location and interaction data with a timetable which records what the child was expected to do during the day, and information about typical activities (e.g., if the child's location is SwimmingPool, the child's activity is probably swimming). From this it creates a series of events which describe the child's lessons and activities, including divergences from what is expected in the timetable. Several messages may be associated with one event. The data analysis infers which events and messages might be most interesting to the child; this is partially based on rules about what children are interested in (e.g., swimming is more interesting than lunch), and partially based on the general principle that unexpected things (divergences from the timetable) are more reportable than expected things. No more than five events are flagged as reportable and only these events are shown in the editing interface.

The editing interface allows children to remove events they do not want to talk about (perhaps for privacy reasons) from the list of reportable events. It also allows children to add messages that express simple opinions about events; i.e., "I liked it" or "I didn't like it". The narration interface allows children to choose a specific event to communicate, which must be one of the ones they selected during the editing phase. Children are encouraged to tell events in temporal (time) order, but this is not mandated and they can deviate from temporal order if they wish. The interfaces use a touch screen but can be scanned with a single switch. Pictorial symbols are used to represent events, annotations, etc.
The natural language generation is fairly simple, since the system deliberately uses simple "child-like" language.

However, the system makes some decisions based on discourse context, including choosing appropriate referring expressions (e.g., pronouns - "it was nice" or "she is nice"), and temporal expressions (e.g., "I then had maths", or "After lunch I had reading").

The system was developed in close collaboration with staff, pupils and parents at a special school. It was trialled with two children, both with severe motor impairments, one of whom is non-speaking and one of whom has cognitive and memory difficulties. The results showed that this technology can both enhance interactive conversation and support storytelling skills and memory sequencing.

We have succeeded in proving the feasibility of using sensor data to support narrative. However, there are several ways in which the system can be improved; the system must be more robust and the inclusion of more child vocabulary, e.g., "it's cool", would be helpful. We were awarded follow-on funding through a Digital Economy Research-in-the-Wild Award to develop a fully-functioning robust prototype version of the system.
Exploitation Route 1) Workshops with clinicians have provided training in ways to support narrative using current voice output communication devices. Clinicians are able to use our research to support their interventions with people with complex disabilities.
2) Developers of assistive technology are able to utilise this research to develop functionality to support narrative.
Sectors Digital/Communication/Information Technologies (including Software),Education,Healthcare
URL http://howwasschooltoday.computing.dundee.ac.uk/
 
Description The research team were able to deliver three successful study days on narrative in collaboration with Communication Matters UK (the UK charity for Augmentative and Alternative Communication). Over 200 participants attended the study days in London, Manchester and Edinburgh. Communication Matters is committed to providing opportunities for professional development and were pleased at the positive reception of the study days and the knock-on dissemination of current research into practice. Staff and pupils at a special school in Scotland benefited from the research collaboration. The school is committed to being at the forefront of technological applications for people with disabilities. Not only did staff and pupils benefit from having researchers in our schools, but they felt they were able to influence the design of new technology, thus ensuring its future effectiveness in the real world. The project benefitted from national and international press and TV publicity, having top billing on regional and national BBC TV News.
First Year Of Impact 2009
Sector Digital/Communication/Information Technologies (including Software),Education,Healthcare
Impact Types Societal,Policy & public services
 
Description Aberdeen - Ehud - HWST Feasability 
Organisation University of Aberdeen
Country United Kingdom 
Sector Academic/University 
PI Contribution Our research team contributed roughly one-half of the research effort in the project.
Collaborator Contribution Their research team contributed roughly one-half of the research effort in the project.
Impact The entire project is an outcome of the collaboration: it was a single project, funded by two separate EPSRC grants. The outcomes recorded here are for EPSRC grant EP/F067151/1. The collaborators (University of Aberdeen) held a similar EPSRC grant EP/F066880/1.
Start Year 2008
 
Description Capability Scotland 
Organisation Capability Scotland
Country United Kingdom 
Sector Charity/Non Profit 
Start Year 2008
 
Description Communication Matters 
Organisation Communication Matters
Country United Kingdom 
Sector Charity/Non Profit 
Start Year 2008
 
Description Dynavox Ltd 
Organisation Dynavox Systems
Country United Kingdom 
Sector Private 
PI Contribution Provided access to new software ideas.
Collaborator Contribution Provided research equipment and collaborated on new software.
Impact Ongoing sharing of new insights.