Adventures in mirror world: Uncovering the cognitive and sensory basis for natural behaviour in virtual reality

Besides their obvious value for entertainment, VR devices are becoming increasingly attractive for the systematic experimental investigation of human perception and behaviour because they promise the possibility of experimental control over environments that approach the complexity of natural settings. If we investigate behaviour or train certain real-world tasks (e.g., in sports or medical training) with the help of VR devices, we expect that the insights we gain or the skills we acquire transfer to real-world settings. Yet, it is currently still unclear if and to what degree behaviour in virtual and natural environments is comparable.

One reason for differences between behaviour in natural and virtual environments is that currently available VR devices simulate environments that are perceptually less complex than their natural counterparts and lack many sources of information that we use to plan and execute our actions in the real world (e.g., missing or inappropriately implemented cues to depth). However, perceptual limitations of currently available VR devices may not be the only source of differences in behaviour. What has so far largely been neglected is how cognitive factors shape behaviour in VR. Particularly, the awareness of interacting with virtual objects can change expectations about the consequence of our actions. For example, while we have clear expectations regarding the immediate consequences of collisions with real objects (e.g., pain or injury and/or damage to the object), collisions with virtual objects usually do not have such direct consequences. Thus, even if VR was perceptually perfect, as long as people are cognisant of its virtual nature, the cognitive effects are likely to alter the way we act in VR (e.g., how cautiously we move). It is therefore essential to understand the exact nature of these cognitive effects.
Another feature of many VR applications that might affect our actions is that users often do not directly interact with real or virtual objects but instead use some device (e.g., a joystick) to manipulate them. In that case, the relation between biomechanical effort and resulting movement differs from their natural relation. This is crucial as we know that natural movements are shaped by biomechanical demands and the biomechanical constraints of our body.
What makes the systematic investigation of the influence of these factors difficult is that due to the limitations of currently available VR devices, it is impossible to disentangle their effects. For example, the effects of the reduced perceptual complexity on movements cannot be easily separated from those resulting from the knowledge of interacting with virtual objects.

Here, we propose a method that allows us to directly compare behaviour in natural and virtual environments independent of the limitations of currently available VR devices. We will simulate a virtual environment by using a powerful but in this kind of research underutilised tool: mirrors. Mirrors are simple and easy to use, yet they are visually the most complete and convincing virtual reality device available. Using mirrors to simulate VR has the advantage that the mirrored environment provides the same visual information as the corresponding natural environment. Therefore, we can investigate behaviour in virtual environments while being able to control the confounding effects of reduced perceptual complexity which is impossible with currently available VR devices. With this simulated virtual reality, we will systematically investigate the three features of VR influencing behaviour (i.e., reduction of perceptual complexity, virtuality of objects and virtuality of movements) using ecologically valid obstacle avoidance and grasping tasks. Our goal is to provide the scientific basis for implementations of VR that are useful for the investigation of human actions and advance knowledge on the interrelationship of perceptual, cognitive and visuomotor processes.