Query-driven Data Acquisition from Web-based Data Sources

The functioning of entities as diverse as enterprises and government agencies depends onobtaining high-quality data.Increasingly these entities depend on external sourcesfor their operational data: critical datais obtained dynamically via web services, is extracted from web pages,or is purchased from third parties. These sources can differ radicallyin their completeness, accuracy, and availability. It is not possible for applications to indexand explore data from each source in advance of querying:there are too many sources, they are too costly to access, and the data in themmay be refreshed constantly. How should data acquisition proceed in such situations?In this project we will develop algorithms for answering queries in the presence of large numbers ofweb-based data sources, sources that may overlap substantially in their datasetsbut have different access restrictions and costs. Our approach will make use of schema information about thedata an application is querying: data format, integrity constraints, and any prior knowledge of costs that maybe available. The core of the project will be algorithms for answering a query by interactively exploring the sources,dynamically pruning out irrelevant or exhausted sources in the process.