Query-Driven Indexing for Peer-to-Peer Text Retrieval

We present a query-driven algorithm for the distributed indexing of large document collections within structured P2P networks. To cope with the bandwidth consumption problem that has been identified as the major issue for the standard distributed approach using single term indexing, we leverage a distributed index that stores top-k document references only for carefully chosen indexing term combinations. In addition, since the number of possible term combinations extracted from a document collection can still be very large, we propose to use query statistics to index only such combination that are indeed frequently requested by the users. Thus, by avoiding the maintenance of superfluous indexing information, we achieve a substantial reduction in bandwidth and storage. A specific activation mechanism is applied to take into account changes in the query distribution, resulting in an efficient, constantly evolving query driven indexing structure. Moreover, our approach facilitates adjusting the indexing load according to the resources provided by the peers in the network. We claim that the size of the index and the generated indexing/retrieval traffic remains manageable even for web-size document collections at a price of a marginal loss in recall for rare queries. Our theoretical analysis and experimental results provide convincing evidence about the feasibility of the query-driven indexing strategy for large scale P2P text retrieval. Furthermore, our experiments confirm that the retrieval performance is only slightly lower than the one obtained with state-of-the-art, centralized query engines, such as Google or Yahoo.