Using Highly Discriminative Keys for Indexing in a Peer-to-Peer Full-Text Retrieval System

Karl Aberer, Fabius Klemm, Vinh Toan Luu, Martin Rajman
2005
Report or working paper

Abstract

Excessive network bandwidth consumption, caused by the transmission of long posting lists, was identified as one of the major bottlenecks for implementing distributed full-text retrieval in a Peer-to-Peer (P2P) architecture. To address this problem we introduce a novel approach to indexing using highly discriminative terms and term sets, which leads to short posting lists and therefore reduces the network traffic by almost one order of magnitude. In addition, we show that retrieval based on discriminative term sets provides a retrieval quality comparable to standard full-text retrieval using TF-IDF ranking. Our indexing scheme is an important improvement towards realistic P2P retrieval systems that opens the opportunity to virtually unlimited scalability well beyond the capacity of today's best centralized Web search engines.

Official source

https://infoscience.epfl.ch/record/52744?ln=en

About this result

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Related concepts

Related publications

, , , ,

Web search engines designed on top of peer-to-peer (P2P) overlay networks show promise to enable attractive search scenarios operating at a large scale. However the design of effective indexing techniques for extremely large document collections still raises a number of open technical challenges. Resource sharing, self-organization, and low maintenance costs are favorable properties of P2P overlays in the perspective of large-scale search, but we also face new problems due to potentially huge bandwidth consumption during both indexing and querying, as well as the unavailability of global document collection statistics. Since a straightforward application of P2P solutions for Web search generates unscalable indexing and search traffic, we propose a novel indexing technique which maintains a global key index in structured P2P overlays. Keys are highly-discriminative terms and term sets that appear in a restricted number of collection documents, thus limiting the size of the global index, while ensuring scalable search cost. Our experimental results show reasonable indexing costs while the retrieval quality is comparable to standard centralized solutions with TF-IDF ranking. Our indexing scheme represents a contribution toward realistic P2P Web search engines that opens the opportunity to virtually unlimited resources, well beyond the capacity of today's best centralized Web search engines.

2005

Beyond term indexing: A P2P framework for Web information retrieval

Karl Aberer, Fabius Klemm, Vinh Toan Luu, Ivana Podnar, Martin Rajman

Web search over peer-to-peer (P2P) networks shows promise to become an alternative to the state-of-the-art search engines since P2P overlays offer means for decentralized search across widely-distributed document collections. However, the design of effective techniques for P2P indexing and retrieval raises a number of technical challenges due to potentially unscalable resource (e.g. bandwidth, storage) consumption. The paper presents a framework for full-text information retrieval in structured P2P networks and introduces a novel retrieval model based on highly discriminative keys—terms and term sets appearing in a restricted number of documents—that ensure efficient and scalable retrieval. Our goal is to design scalable techniques for building a global key index in structured P2P overlays for large document collections. We present experimental results that show acceptable indexing and retrieval costs while the retrieval quality is comparable to standard centralized solutions with BM25 relevance computation scheme.

2006

Towards a Foundation Model for Peer-to-Peer Web Information

Karl Aberer, Jie Wu

We find that there lacks a formal algebraic framework in the research area of P2P information retrieval. Such a formal framework is indispensable and pivotal. On the one hand, no solid theoretical foundation can explain why the results of many ad-hoc commercial ranking algorithms are good or bad; on the other hand, in some cases, it might be preferable in a decentralized search system to use different retrieval models and different ranking algorithms at the same time since each separate document collection and each peer might have different characteristics. Thus it is important to have a common framework, where we can capture the individual models and describe the overall behavior. Such a framework characterizes notions like ranking, relevance feedback, rank aggregation and combination, etc. and necessary operations. In this paper, we define such a foundation model for P2P information retrieval, where data models for rankings from the perspectives of both the end user's search needs and the search member server's contextual rankings are provided, and a new ranking aggregation language is proposed to perform ranking combination operations. A case study is included to demonstrate how the foundation model and the ranking aggregation language could be applied in real ranking problems of P2P Web search systems.

2004