ADAPTER: an Adaptive Particle Data Format

Increasing amounts of data are being produced by computational science simulations and scientific measurements. This deluge of data is slowing the speed at which new scientific knowledge can be discovered as scientists struggle to extract new knowledge from massive multi-Terabyte data sets.The ADAPTER project aims to structure multi-Terabyte data sets so that scientists can focus on the important aspects directly relevant to the meaning they are trying to extract. The project will produce a new indexed data format that will enable rapid scientific queries on subsets of the data at low as well as high resolutions. ADAPTER will adapt to the structure in the data so that scientific effort can be focussed on important structure. In addition while simulations may be undertaken on massive international HEC computing systems ADAPTER data sets will enable smaller campus-based and desk-side HPC systems to be adapted to the task of analyzing the data.We will demonstrate significantly faster analysis of cosmological particle simulation data sets such as the 20Terabytes Millennium Simulation, where the hunt is on for structures likely to contain galaxies hidden in the data. In addition ADAPTER will be designed to support future simulations which are expected to generate datasets of up to 500Terabytes in size.Even at this early stage the ADAPTER project has attracted interest and support from scientists around the world.