Perception and Saliency in Augmented Reality
Lead Research Organisation:
University of Bath
Department Name: Computer Science
Abstract
Augmented reality (AR) is the direct or indirect view of a real world environment that has been modified, added to, or enhanced by computer-generated sensory input. Typically taking the form of visual additions and effects to the physical environment, AR is a promising and growing research area incorporating human-computer interaction, computer vision and machine intelligence.
Humans are predominantly visual creatures, relying on their sight as their primary means of information gathering. As such, much of our technologies are dominated by separate visual displays, which ultimately detract from our perception of the world around us. AR has the potential to shift visual information and interaction to the world around us, allowing us to stay engaged with our environment while enhancing the experience with extra sensory input. Further, AR potentially provides a natural and elegant way of displaying information that relates to the environment, or regions of interest within it, to the user.
AR, however, is not an new concept. There has been plenty of research and implementation within both academia and industry. However these solutions have often been underwhelming, impractical, or both. AR needs to enhance a user's perception of their environment without detracting from it in other areas. Three obvious requirements of a good AR system are:
1) Robust and accurate augmentation
2) Dynamic augmentation
3) Personalised, appropriate or context driven augmentation
Robust and accurate augmentations are key to user engagement with a system. A system that is annoying and doesn't display things how they ideally should is useless. Dynamic augmentation allows variation in experience and information i.e visually displaying more detailed information about an object as you move closer to it. This again is important to engagement, and allows for a flexible system. Personalisation, appropriateness and context awareness allows the system to be relevant to the user. What does the user see as important in their environment? This question obviously affects what, and how, you augment and user's environment and experience.
The proposed work throughout this PhD is threefold. Robust and accurate augmentation can be considered a problem of localisation, both of the person and objects of interest. Study, research and implementation of various hardware (gyroscopes, accelerometers, depth cameras) and computer vision techniques (object recognition, visual SLAM) is likely to reveal solutions. Study of the human visual system, perception and how to computationally simulate it (saliency) with regards to context is likely to provide insight into how to personalise AR for the user's needs. Finally, study of AR systems and how people interact and engage with them will reveal how best to design AR systems to be useful for specific environments and everyday life in general.
Humans are predominantly visual creatures, relying on their sight as their primary means of information gathering. As such, much of our technologies are dominated by separate visual displays, which ultimately detract from our perception of the world around us. AR has the potential to shift visual information and interaction to the world around us, allowing us to stay engaged with our environment while enhancing the experience with extra sensory input. Further, AR potentially provides a natural and elegant way of displaying information that relates to the environment, or regions of interest within it, to the user.
AR, however, is not an new concept. There has been plenty of research and implementation within both academia and industry. However these solutions have often been underwhelming, impractical, or both. AR needs to enhance a user's perception of their environment without detracting from it in other areas. Three obvious requirements of a good AR system are:
1) Robust and accurate augmentation
2) Dynamic augmentation
3) Personalised, appropriate or context driven augmentation
Robust and accurate augmentations are key to user engagement with a system. A system that is annoying and doesn't display things how they ideally should is useless. Dynamic augmentation allows variation in experience and information i.e visually displaying more detailed information about an object as you move closer to it. This again is important to engagement, and allows for a flexible system. Personalisation, appropriateness and context awareness allows the system to be relevant to the user. What does the user see as important in their environment? This question obviously affects what, and how, you augment and user's environment and experience.
The proposed work throughout this PhD is threefold. Robust and accurate augmentation can be considered a problem of localisation, both of the person and objects of interest. Study, research and implementation of various hardware (gyroscopes, accelerometers, depth cameras) and computer vision techniques (object recognition, visual SLAM) is likely to reveal solutions. Study of the human visual system, perception and how to computationally simulate it (saliency) with regards to context is likely to provide insight into how to personalise AR for the user's needs. Finally, study of AR systems and how people interact and engage with them will reveal how best to design AR systems to be useful for specific environments and everyday life in general.
Organisations
Studentship Projects
| Project Reference | Relationship | Related To | Start | End | Student Name |
|---|---|---|---|---|---|
| EP/N509589/1 | 01/10/2016 | 30/09/2021 | |||
| 1789379 | Studentship | EP/N509589/1 | 01/10/2016 | 31/08/2020 | Jacob Elliott Hadnett-Hunter |
| Description | Throughout this project we have made several key discoveries and contributions to the research field. We have proposed a new methodological approach to studying human visual attention in dynamic virtual environments, with the ability to draw parallels from these studies to real life. This has significant impact on the psychophysics and attention literature. By eye tracking participants as they navigate virtual 3D worlds, we can replicate scenarios and data collection that is impossible in real like. Further, coupled with feature and saliency based analysis of this data, we can ask important questions suchs as: "Where do people allocate their attention depending on scene and task?", "What things draw their attention?", "How can we alter their attention?". Our first published piece of work, "The effect of task on visual attention in interactive virtual environments" [1] demonstrated the how the task that someone is undergoing effect the allocation of their visual attention in virtual worlds. Further to this, this publication also helped to demonstrate the usefulness of virtual environments for vision science, something which we are planning to extend in the coming year. [1] Hadnett-Hunter, Jacob, et al. "The effect of task on visual attention in interactive virtual environments." ACM Transactions on Applied Perception (TAP) 16.3 (2019): 1-17. |
| Exploitation Route | Our research allows for better understanding of human visual attention in virtual media. The methodology and results of our research can be used to produce better digital material for people to engage with. Games, simulations and 3D stereoscopic film can all benefit from knowing where and why a person might look. Our research provides this insight. |
| Sectors | Digital/Communication/Information Technologies (including Software) |
| Description | Presentation of "The effect of task on visual attention in interactive virtual environments" at SAP2019, Barcelona |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Present our Transactions on Applied Perception paper (Hadnett-Hunter, Jacob, et al. "The effect of task on visual attention in interactive virtual environments." ACM Transactions on Applied Perception (TAP) 16.3 (2019): 1-17.) at the Symposium of Applied Perception 2019, Barcelona. Many (50+) researchers in vision science, graphics, computer vision, psycho physics etc attended and view my presentation. Q/A resulted in a good discussion of impacts and further work that the community would be interested. |
| Year(s) Of Engagement Activity | 2019 |
| URL | https://sap.acm.org/2019/schedule.php |
| Description | Visual Attention in Art - Holburne Museum |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | Local |
| Primary Audience | Public/other audiences |
| Results and Impact | On two separate occasions we set up an eye tracking stool at the Holburne Museum, Bath in order to demonstrate to people their own visual attention of art. One session was for the general public as part of the "Seurat to Riley" exhibition, and another was with local school children. For the demonstration, people would come up and I would explain to them about eye tracking and then they could view a series of paintings on screen while they were eye tracked. Afterwards, I would show them visualisations of their gaze data, and chat with them about visual attention, research, potential impacts etc. |
| Year(s) Of Engagement Activity | 2017,2018 |