Autonomic Data Management for Very Large Dataset Visualization

Never before in history have we had such capability for generating, collecting and storing digital data. Data repositories at terabyte level are becoming commonplace in many applications, including bioinformatics, medicine, remote sensing and nano-technology. In some applications, such as network traffic visualization and video visualization, we are encountering the scenario that dynamic data streams are almost temporally unbounded. Many visualization tasks are evolving into visual data mining processes.The main scientific challenge to be tackled in this proposed collaborative project is how to manage visualization tasks that involve very large datasets which must reside on one or a few remote data servers for various reasons, including excessive space requirements, data sharing provision, dynamic data generation, and data security. Because of the rapid developments of data capture and data generation technologies (e.g., web-based data collection and Grid computing), the management of such visualization over a complex computing and communication infrastructure is becoming a bottleneck to interactive visual data mining. It is not possible to device a uniform approach to suit various requirements nor cost-effective to place the burden of complex data management upon users.One emerging strategy for developing complex computing infrastructure is autonomic computing, which seeks inspiration in self-adaptive biological systems and self-governing social and economic systems. In a recent survey coordinated by the PI, the concept of autonomic computing was first brought into the context of visualization, and it was suggested that the concept can be deployed in a visualization infrastructure in an evolutionary manner. It is desirable to investigate into an autonomic approach to the management of very large datasets in visualization, to design and prototype knowledge-based framework for capturing dynamic system information and users' experiences, and to develop adaptive algorithms that can utilize such information dynamically for data management in visualization.The aim of this research is to develop an autonomic approach to the management of very large dataset visualization. The joint research programme will be carried out primarily by the PI, Professor Min Chen, and his PhD student, Mr. David Chisnall, in collaboration with Professor Charles Hansen and his research team in University of Utah.The main resources for conducting this joint programme is in place with a current PhD studentship held by Mr. Chisnall (January 2004 - December 2007). Some work on O1, O2, and O3 has been carried out. We therefore do not seek any funding for staffing and equipment costs. The funding sought for this project is for covering travelling and subsistence cost, as face-to-face meetings and short-term visits are indispensable in such close collaboration involving intensive algorithm design and software development, as well as direct access to computing infrastructure, datasets, and scientific users in Utah.