Multimodal, Negotiated Interaction in Mobile Scenarios
Lead Research Organisation:
Swansea University
Department Name: College of Science
Abstract
Our proposal is to investigate an alternative means of allowing users to interact with content and services in their environment such that the actions they make, movements, gestures, etc., and feedback they receive are continuous, with the user and system negotiating their interactions in a fluid, dynamic way. We believe the appropriate comparison would be dancing, rather than the current command & control metaphor. When someone dances with a partner there is a soft ebb and flow of control; sometimes one person leads, sometimes the other, this changing fluidly as they dance. We are proposing a similar interaction between a user and computer, where sometimes the user leads and at other times the computer according to the context of the interaction. This contrasts with most current approaches where one agent, be it the human or the computer, pre-empts the other and where most interaction is driven by events and proceeds to varying degrees in rigid, over specified waysExample scenario: Exploring a Digitally Enriched EnvironmentAn example of how this approach could be used is that of location-aware information acquisition while walking in a town centre. You might feel a 'tick' on your phone's vibration motor, making you aware that there is information available about something in your environment. Your rich context understanding abilities would tell you how likely this 'tick' was to be of interest, if you ignore the cue and walk on, the negotiation would end there and then. If you are curious, you might gesture with the phone at likely targets in your surroundings, and get a response from several of them. If you are further intrigued, you may continue to interact with these potential targets, possibly moving from the vibro-tactile to an audio display, gaining information by an active exploration of the environment, something we have evolved to do naturally. The user explores the possibilities in the situation by directly engaging (probing or playing) with it, being able to move at will through the space of possibilities, gaining more and more insight during the interaction. The multimodal feedback provided both encodes the system's current interpretation of the user's intention (e.g. moving towards a target) and the probability of the target meeting the user's needs. After working through combinations of vibration and audio, if the joint dynamics of information source, and user continue to intertwine, the display of the mobile device might be used for full details. This example shows a 'schedule' of modalities, and illustrates the negotiation process in a practical and commercially interesting example.
People |
ORCID iD |
Matt Jones (Principal Investigator) |
Publications

Jones M
(2011)
Classic and Alternative Mobile Search A Review and Agenda
in International Journal of Mobile Human Computer Interaction

Robinson S
(2010)
"I did it my way"

Robinson S
(2009)
Exploring casual point-and-tilt interactions for mobile geo-blogging
in Personal and Ubiquitous Computing

Robinson S
(2010)
Heads-up engagement with the real world

Robinson S
(2009)
Sweep-Shake

Robinson S
(2011)
Navigation your way: from spontaneous independent exploration to dynamic social journeys
in Personal and Ubiquitous Computing

Robinson S
(2008)
Point-to-GeoBlog

S Robinson
(2009)
Sweep-Shake: Finding Digital Resources in Physical Environments

Williamson J
(2010)
Social gravity

Wilson M
(2010)
Pico-ing into the future of mobile projector phones
Description | Novel ways to interact with mobile devices and services through: gesture, haptic feedback, sound and movement. |
Exploitation Route | Range of concepts and implementations that can be built into mobile devices and services. |
Sectors | Creative Economy,Digital/Communication/Information Technologies (including Software),Leisure Activities, including Sports, Recreation and Tourism,Culture, Heritage, Museums and Collections |
URL | http://cs.swan.ac.uk/negotiatedinteraction/index.php |