Multiple Scale and Multimodal Data and Information Fusion in Human Sensory Discrimination

This is an inter-disciplinary project involving combined psychophysical studies of human uni- and cross-modal sensory discrimination and scene analysis, along with computational modeling of the data using multi-dimensional time series analysis. The psychophysical experiments will provide novel information on the use of multiple haptic inputs as a function of type and relative positions of receptors being stimulated, as well as information on how concurrent haptic and visual input is integrated. The computational studies will evaluate whether copula-based wavelet analyses are able to simulate human discrimination performance, with information being combined immediately across sensory inputs. As an alternative, this will be compared with approaches in which time series (including wavelet-based) analyses are conducted for each input before integration takes place across inputs. The work will provide a detailed account of how sensory integration operates within and across modalities, which can subsequently be used to guide the construction of artificial sensory systems.

In biological systems, multiple sensory inputs provide the basis for our ability to discriminate and recognize objects / an ability that is markedly sensitive to correlations between the sensory signals. However, these inputs have contrasting transmission and sampling rates, and are derived at different resolutions, providing a challenge for attempts to model the integration process. This project will attempt to model novel data on human sensory discrimination and object recognition using mathematical approaches from computer science and engineering based on multi-dimensional time series analyses. Initially psychophysical studies will be conducted in which new information is gained from haptic and combined haptic-visual discriminations. The experiments will require either discrimination of a target signal (sensitive to data integration across channels) or same-different matching of objects (sensitive to information integration), and performance with correlated and uncorrelated inputs will be tested as the temporal interval between signals is varied. The modeling work will use a new copula-based approach to wavelet analysis, which binds the joint cumulative distribution function of signals to the marginal cumulative distribution functions, potentially enabling correlated input patterns to be integrated directly across different spatial and temporal resolutions. This approach will be contrasted with a within-channel integration process, where wavelet-based time series analysis is performed with data from one sensor before there is integration across sensors. The work will provide new data on human sensory discrimination abilities, whilst evaluating the best way to model such data mathematically. In the longer-term the model will help to guide the construction of artificial sensory systems, based on biological capabilities.