Kinetic User Interfaces and Multiuser 3D Virtual Worlds for Older People

Lead Research Organisation: University of Kent
Department Name: Sch of Engineering & Digital Arts

Abstract

Older people may experience changes with age, such as a decline in physical and cognitive capabilities, which could result in deterioration of their social relationships and quality of life. It is important to help them to maintain independence and delay the need for institutionalised care. It is therefore crucial to find novel ways that motivate them to have a more active life. One area of such possibility is 3D virtual worlds (VW), which provide a creative and social space for users to engage in various activities.

In fact, virtual worlds, such as Second Life, have increasingly showed promise in healthcare. There are for instance a variety of healthcare support groups for topics such as mental health, disabilities, depression and cancer in virtual worlds. Most importantly, it is claimed that virtual worlds could help chronically ill patients and disabled users by allowing them to temporarily leave their disabilities behind and participate in social activities and better express their ideas and creativity through association with groups in virtual worlds. Studies suggest that frequent engagement in creative activity, such as computer game playing, might be helpful in reducing the risk of Alzheimer's disease and combating serious depression among older people.

However, a key obstacle when implementing virtual worlds for healthy ageing is the adoption of conventional user interfaces (such as the keyboard and mouse) which are often not well suited for older people. For instance, navigation (moving around in virtual world) and object manipulation (such as selecting and positioning a virtual object) pose serious usability issues even for younger users who are reasonably familiar with the computer. Another potential issue concerns the design of non-verbal cues for online communication through the use of avatars (graphical virtual representation of the user). Non-verbal cues are important in human communication. In virtual world, users can express themselves through the use of avatar gestures (such as waving, bowing), usually by pressing certain shortcut keys on the keyboard. An important design issue therefore lies in finding a natural way to allow for more expressive social interaction with nonverbal cues.

In recent years, there has been an emergence of a new user interface paradigm known as Natural User Interfaces (NUI). Instead of using conventional devices such as the keyboard and mouse, users interact with their computers using gestures, speech, and thoughts. One type of NUI, known as Kinetic User Interfaces (KUI) allows users to interact with the computer through body motions. Examples of such devices include Microsoft Xbox Kinect and Nintendo Wii. Such relatively natural ways to interact with technologies make it easier to learn, thus lowering the barrier of engagement.

Therefore, the proposed research programme aims to conduct a detailed investigation on how Kinetic User Interfaces (KUI) can be employed effectively to design innovative 3D virtual worlds accessible to older people. Older people's behaviour will be studied in depth to inform to 3D design solutions for this specific population. Prototypes will be developed and evaluated to assess the effectiveness of KUI-based 3D design in virtual world. Addressing these issues, we ultimately aim to make 3D technology more accessible and engaging for older people, thus motivate them to participate in active life style through this technology.

Planned Impact

The ultimate outcome of the research will be to motivate older people to engage in active lifestyle through the use of 3D virtual worlds (VW). The direct involvement of the users (older people) and AgeUK is integral to the project to ensure maximum impacts. Older people will be involved directly to elicit user requirements through observational studies and participatory design workshop. Officers from AgeUK Canterbury will provide their expertise in the design of the prototypes, as well as helping to manage participant recruitment.

Furthermore, we will work with Silverfit (http://www.silverfit.nl/), which will share design expertise and technical skills regarding the use of KUI technology for older people. Silverfit has substantial experience in developing rehabilitation technology based on physiotherapy guidelines and their products are used in care homes and therapy centres around the globe. Silverfit is very keen on the use of 3D technology in their work. For instance, they are exploring the use of 3D environment which is familiar to the users (e.g. kitchen) in occupational therapy. The results of the proposed project can be utilised by Silverfit to further develop the prototype system, with the aim to eventually install such system in AgeUK centres and various care homes.

The results will highlight key design heuristics and specific guidelines and will be used to direct future development of VW or 3D technology. It dovetails with AgeUK's aim to help older people age healthily by engaging with active life socially, mentally and physically. The project will also serve as an exemplar to other communities in the UK and internationally as to how we can motivate older people, or indeed those with very little computer experience to engage with 3D technology, which could have various active lifestyle applications.

Given the unique characteristics of VW, impacts can be extended beyond healthy ageing. For instance, the PI is exploring collaboration with Cranfield Univ. and related industries (e.g. BAE system) to look into such technologies in training in the area of maintenance engineering. The project is also likely to be of interest to culture and heritage organisations such as museums. The PI and colleagues have been working with local museums in Kent (e.g. Powell Cotton Museum and Turner Contemporary) and there have been substantial interests in using KUI-based VW to engage visitors with cultural artefacts in an immersive experience.

Although the design of VW is likely to be varied in different domains, the project outcomes can provide an indication on how intuitive, accessible and engaging user interfaces could be designed. The advantage of accessibility studies such as this is that by making the interfaces accessible to older people often with little computer experience, one also improves the general usability of the system. It is especially important for 3D technology, where many average computer users still have difficulty using them.

The PI will be responsible for disseminating research to relevant audiences through passing on relevant publications and through liaison with the press offices of the University and EPSRC. This will be particularly relevant for communicating findings to the general public. Through liaison with AgeUK Canterbury, we will communicate the findings to a wider audience in AgeUK. The PI will also actively communicate the results and research outcomes with relevant audiences including museums, East Kent Hospital Univ. Trust (our collaborator in health-related research), care homes (e.g. our partner Graham Care Group), technology developers (Silverfit), educational institutions through our wide collaboration networks. For this purpose, we will develop digital (website, documentary developed by students at Univ. of Kent as part of their degree) and offline materials (e.g. leaflet, poster) to engage not only industries and businesses, but also the general public.

Publications

10 25 50
 
Description The care of dementia (and elderly care in general) is becoming a key agenda in health care research. Therefore, there is a strong interest in finding ways to assist the care of older people. Increasingly, the multidisciplinary research community is looking into ways to design technology to support specifically dementia care. Virtual world (VW) technologies have great potentials in enhancing dementia care. The high fidelity imagery generated in VW could create "living experiences", experiences which are interactive, engaging and immersive for people with dementia. Reliving past experiences could help stimulate memories, provide positive emotions and enhance their sense of self.



Four versions of gesture-based 3D virtual world prototypes have been developed and tested iteratively in 2 Avante care homes, AgeUK Canterbury and Alzheimer's Society (Hawkinge) over the course of 1 year.



Version 1: Reminiscence Room Prototype

The first prototype presented the user with an avatar placed in a virtual room with objects from the past (old posters, magazines, TVs, books). A radio was also included in the virtual room playing music of that period. Users' movements were captured by Microsoft Kinect and mapped to their virtual avatar.



Version 2: Virtual Tour Prototype

The second version of the prototype was created based on the idea of users being taken along for a virtual tour through a pre-determined path. For instance, The user would tour around the river by doing a "rowing" motion with their arms, which would propel the boat forward along the path. See an example: http://www.youtube.com/watch?v=bflFQOPc9YI



Version 3: Gardening Prototype

In this prototype, the participant would work together with the caregivers to design a virtual garden. 20 objects were available for selection including nine types of flowers, four types of trees and seven types of vegetables. An Android-based tablet computer was used to allow the caregiver/residents to select the vegetation to plant in the virtual garden



Version 4: Tangible User Interfaces

This version is similar to version 3, but instead of using a tablet, the user would put a plastic flower (tagged with NFC) on a platform in order to plant a flower in the virtual garden. See an example: http://www.youtube.com/watch?v=Q3jjFjIJJRU





Through the study, we have explored in depth how VW technology was used by people with dementia and their caregivers in long-term care. Testing and revising a number of prototypes also gave insights related to design issues. Each prototype had different benefits for difference group of users and it could not be said that one is better than the other. We are also not advocating replacing physical activities with only virtual activities. Instead, we believe that VW can serve as follows:

• Providing interactive experiences to promote a continuing selfhood which may have been lost due to dementia and/or moving to long-term care facilities.



• Providing stimulation and engagement and to promote a feel good factor. Negative memories could be mitigated by playful design (e.g. "wonderland approach").



• Being in a "place" that allows staff and residents to foster personal relationship and trust, which may improve care.





Although it appeared from the observation that VWs worked with many residents to a certain extent, there were incidents where they did not work well. We highlight some problems we encountered, which could direct future design and research work:

• Excessive physical motion: for some residents, continuous use of Kinect caused fatigue. Some residents kept on performing the physical motions and would not stop until being told by the carers. In addition, many residents were confined to wheelchairs and this did not permit them to use the virtual room prototype.



• Personally, irrelevant places, objects and activities: The virtual garden and park prototypes were especially successful among female residents but less so with the male residents who preferred other activities such as driving or sports. This underlines a key limitation of the wider use of VW in care home as creating different virtual environment is rather costly at the moment.



• In some cases, negative memories were triggered from the VW. Such cases required the care staff to redirect their attention back to the positive aspect of the virtual activity.



• Although some care staff was very keen to try the technology, others were less enthusiastic as the technology was perceived as adding extra workload to their already busy schedule.



• Most of the people with dementia were able to interact with the prototypes with relative ease. However, we observed a few cases (those with more severe cognitive declines) where the residents were unable to perform the simple tasks on touch screen or using gesture. In these cases, their visual and audio capabilities have been significantly reduced and often the caregivers would guide them by physically touching them (especially their shoulders and hands) to get their attention.
Exploitation Route - use in Alzhemier's Society day centre and home visit

- use in care home setting (this will require further automation, including using intelligent virtual avatar)
The non-academic partners we work with (especially Alzheimer's Society) is very keen to implement this technology in their day centre. They are also very keen to explore the possibility to bring this technology to their clients during home visit. However, the prototype is still in its early stage, and further work needs to be done to turn it into a product. A new PhD student has been recruited at University of Kent under the supervision of the PI to further explore the exploitation route, working closely with Alzheimer's Society
Sectors Digital/Communication/Information Technologies (including Software)

URL http://jimmybbq.wordpress.com/virtual-world-and-dementia/
 
Description The research has resulted in a close collaboration with Alzheimer's Society, Hawkinge. The final version of the virtual world prototype was presented to the managers, and a formal collaboration has been formed to test the prototype with their clients. The findings have also led to a new PhD project looking into how people with dementia perceive and interact with 3D models in collaboration with Alzheimer's Society, Hawkinge. On a perhaps slightly less related topic, some of the experimental technologies during the research study were presented to the Royal Society of Chemistry in London in 2013, and it has gained some interests in how the technologies can be used for Chemistry education. Currently we have completed a new NHS approved study with St Andrew's Hospital using VR on dementia patients. We anticipate this to be deployed more widely in the hospital.
Sector Education,Healthcare,Other
Impact Types Societal

 
Description AgeUK 
Organisation Age UK
Department Age UK Canterbury
Country United Kingdom 
Sector Charity/Non Profit 
PI Contribution the research team has provided prototypes of new virtual world technologies for people with dementia
Collaborator Contribution The partner has contributed their expertise in elderly care
Impact A conference paper has been published. The collaboration involves HCI and elderly care
Start Year 2011
 
Description Alzheimer's Society 
Organisation Alzheimer's Society
Country United Kingdom 
Sector Charity/Non Profit 
PI Contribution The research team has provided prototypes of new virtual world technology which could benefit their clients
Collaborator Contribution They have provided space and helped identified participants in the study
Impact a conference paper has been published. it involves HCI and dementia care
Start Year 2012
 
Description Avente care 
Organisation Avante Care and Support
Country United Kingdom 
Sector Charity/Non Profit 
PI Contribution the research team has provided prototypes of new virtual world technology which may benefit their clients
Collaborator Contribution the partner has provided space and helped identified participants for the study
Impact A conference paper has been published. It involves HCI and dementia care
Start Year 2012
 
Description Silverfit 
Organisation Silverfit
Country Netherlands 
Sector Private 
PI Contribution the research partner contributed by providing insights into designing virtual worlds for people with dementia
Collaborator Contribution the partner has contributed their expertise in designing technology for older people
Impact a conference paper has been published it is not a multi-disciplinary collaboration
Start Year 2012
 
Title Virtual world prototype 
Description Several versions of the 3D virtual world (VW) prototypes have been created to support life engagement for people with dementia in long-term care. Version 1: Reminiscence Room Prototype The first prototype presented the user with an avatar placed in a virtual room with objects from the past (old posters, magazines, TVs, books). A radio was also included in the virtual room playing music of that period. Users' movements were captured by Microsoft Kinect and mapped to their virtual avatar. Version 2: Virtual Tour Prototype The second version of the prototype was created based on the idea of users being taken along for a virtual tour through a pre-determined path. For instance, The user would tour around the river by doing a "rowing" motion with their arms, which would propel the boat forward along the path. See an example: http://www.youtube.com/watch?v=bflFQOPc9YI Version 3: Gardening Prototype In this prototype, the participant would work together with the caregivers to design a virtual garden. 20 objects were available for selection including nine types of flowers, four types of trees and seven types of vegetables. An Android-based tablet computer was used to allow the caregiver/residents to select the vegetation to plant in the virtual garden Version 4: Tangible User Interfaces This version is similar to version 3, but instead of using a tablet, the user would put a plastic flower (tagged with NFC) on a platform in order to plant a flower in the virtual garden. See an example: http://www.youtube.com/watch?v=Q3jjFjIJJRU 
Type Of Technology Software 
Year Produced 2013 
Impact Testing and revising a number of prototypes also gave insights related to design issues. Each prototype had different benefits for difference group of users and it could not be said that one is better than the other. We are also not advocating replacing physical activities with only virtual activities. Instead, we believe that VW can provide interactive experiences to promote a continuing selfhood which may have been lost due to dementia and/or moving to long-term care facilities. It can also provide stimulation and engagement to promote a feel good factor. Negative memories could be mitigated by playful design. VW could also be a "place" that allows staff and residents to foster personal relationship and trust, which may improve care. 
 
Description Talk 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact 100+ audience from the public attended a talk organised at University of Kent

The audience expressed deep interests in this research work
Year(s) Of Engagement Activity 2014
 
Description Workshop 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Professional Practitioners
Results and Impact 20-30 NHS practitioners, care workers, software developers attended a workshop organised at University of Kent

The workshop participants became more aware of the problem of dementia and how technologies could potentially help in addressing some of the problems
Year(s) Of Engagement Activity 2012