Integrating visual and semantic descriptions for effective, flexible and user-friendly image retrieval

Effectively managing a large collection of multimedia documents is a challenge, addressed by many disciplines from signal processing through database systems to artificial intelligence and interaction design. The problems to be solved have rarely been considered together. We propose a series of novel solutions for: the system architecture, the document content characterization, the retrieval methodology, and the user interaction schemes. We propose and describe a communicating components architecture using a new open and flexible protocol for messaging. Its foundation, the multimedia retrieval markup language (mrml) is specified with examples of the benefits of this approach. The general multimedia application domain is restricted mainly to image documents that carry semantic annotation like captions or meta-data. The integration of perceptual and semantic content descriptions into a single retrieval structure is the principal contribution of our work. The proposed method allows for improved effectiveness, augmented functionality and high flexibility for the extension to other media types like audio or video. The relationships among the two different representational characteristics can be exploited for more effective retrieval and other tasks like semi-automatic image annotation or illustration of semantic concepts. The final contribution is the study and implementation of user interaction metaphors that are intuitive and attempt to further bridge the gap between user's and system's notion of relevance. A rich set of information need formulation tools is proposed to users of varying skill and of varying requirements. Merging query specification with result visualization and browsing, using interactively explorable search spaces, offers a single access point to most retrieval tasks.

Exploitation de connaissances sémantiques externes dans les représentations vectorielles en recherche documentaire

Florian Seydoux

The work presented in this thesis deals with several problems met in information retrieval (IR), task which one can summarise as identifying, in a collection of "documents", a subset of documents carrying a sought information, i.e.. relevant for a request expressed by a user. In the case of textual documents, to which we limited ourselves within the framework of this thesis, a significant part of the difficulty lies in ambiguity inherent to human languages. The interaction with the user is also approached in our work, by studying a tool enabling a natural language access to a database. Finally, some techniques which permit the visualisation of large collections of documents are also presented. In this document we first of all describe the principal models of IR by highlighting the relations which exist with some manual technics of IR and document retrieval, developed during the past centuries. We present the principle of document indexing, allowing us to represent documents in a multidimensional space, and the use of this representation by a vectorial model. After having reviewed the principal improvements made these last years with vectorial research systems, including the preprocessings of collections, the indexing mechanism and measurements of similarities between documents, we detail some recent usecases of additional semantic resources (semantic dictionaries, thesaurus, networks, ontologies) reported in scientific literature for the indexing task. We then present more in detail the semantic indexing principle of textual documents by using a thesaurus, consisting in integrating in the document's representation space at least part of the informational contents of hierarchical semantic resources. We propose a general framework allowing us to describe and position various possible techniques to carry out the semantic indexing by adapting, if possible, the specificity of the descriptions resulting from the semantic resources to the data to be represented. We use this framework to describe three families of criteria usable for semantic indexing, each one having its own characteristics. For each of these families, we give the specific algorithms allowing the computation of the criteria. The first two families allow us to consider several criteria already known in feature selection. Moreover we show that, unfortunately, many of these criteria are in fact not very effective for the considered task. The third family allows us to introduce a completely new criterion, the Minimum Redundancy Cut criterion (MRC), built on the basis of the information theory and allowing us to obtain index terms having a probability of occurrence in the collection of documents as well balanced as possible. Finally, we treat the case of semantic index independent of the data (statically choosen), allowing a parameterisation of the level of generality of the index terms. Some of the criteria suggested for semantic indexing has been empirically evaluated. To judge their relevance, we used a well known vectorial system (the Smart IR system) and measured the performances of IR obtained with various reference collections. Those collections was indexed on the basis of the studied criterion, by taking into account the strongly structuring semantic relation of hyper/hyponymy ("is-a" relation), given by two different semantic resources. By comparing results obtained with the performances of a traditional indexing (using the lemmas of the words as representation space), we can show on one hand the relevance of the semantic indexings (in RD) and on the other hand the quality of the proposed criterion (MRC). Concerning man-machine interaction, we present a general outline allowing to build in a relatively fast and systematic way systems with mixed initiative, giving the human user a large (and natural) latitude in the control of the dialogue. This outline is usable in typical database research-task applications (where the database is hidden to the user, but the latter knows exactly which information they wish to find) as well as advice-task applications, for which the users does not necessarily have a precise idea of their needs, and uses the system not only for specifing their wishes, but also a set of propositions as a final result. We particularly stress the techniques allowing us to obtain a robust system, able to deal with speech recognizer failures. Concerning the visualisation of large textual data collections, we present an application of the correspondences analysis (allowing to highlight similarities and oppositions for various groups of entity, built on the basis of additional features present in the DB) to the case of patents data. In addition, we propose a method (based on the bootstrap replication principle) allowing us to determine a confidence interval for relative positionings of various groups, thus permit to immediately judge the reliability of the visually apparent similarities or oppositions.

EPFL2006

Thematic Annotation: extracting concepts out of documents

Martin Rajman

Semantic document annotation may be useful for many tasks. In particular, in the framework of the MDM project(http://www.issco.unige.ch/projects/im2/mdm/), topical annotation -- i.e. the annotation of document segments with tags identifying the topics discussed in the segments -- is used to enhance the retrieval of multimodal meeting records. Indeed, with such an annotation, meeting retrieval can integrate topics in the search criteria offered to the users. Contrarily to standard approaches to topic annotation, the technique used in this work does not centraly rely on some sort of -- possibly statistical -- keyword extraction. In fact, the proposed annotation algorithm uses a large scale semantic database -- the EDR Electronic Dictionary(http://www.iijnet.or.jp/edr/) -- that provides a concept hierarchy based on hyponym and hypernym relations.This concept hierarchy is used to generate a synthetic representation of the document by aggregating the words present in topically homogeneous document segments into a set of concepts preserving the document's content. The identification of the topically homogeneous segments -- often called Text Tiling -- is performed to ease the computation as the algorithm will work on smaller text fragments. In addition, it is believed to improve the precision of the extraction as it is performed on topically homogeneous segments. For this task, a standard techniques -- proposed by \cite{richmond97detecting} -- relying on similarity computation based on vector space representations have been implemented. Hence, the main challenge in the project was to create a novel topic identification algorithm, based on the available semantic resource, that produces good results when applied on the automatically generated segments. This new extraction technique uses an unexplored approach to topic selection. Instead of using semantic similarity measures based on a semantic resource, the later is processed to extract the part of the conceptual hierarchy relevant to the document content. Then this conceptual hierarchy is searched to extract the most interesting set of concepts to represent the topics discussed in the document. Notice that this algorithm is able to extract generic concept that are not directly present in the document. The segmentation algorithm was evaluated on the Reuters corpus, composed of 806'791 news items. These items were aggregated to construct a virtual document where the algorithm had to detect boundaries. These automatically generated segments were then compared to the initial news items and a metric has been developed to evaluate the accuracy of the algorithm. The proposed approach for topic extraction was experimentally tested and evaluated on a database of 238 documents corresponding to bibliographic descriptions extracted from the INSPEC database(http://www.iee.org/publish/inspec/). A novel evaluation metric was designed to take into account the fact that the topics associated with the INSPEC descriptions -- taken as the golden truth for the evaluation -- were not directly produced based on the EDR dictionary, and therefore needed to be approximated by the available entries. Alltogether, the combination of existing document segmentation methods -- i.e text tiling -- with novel topic identification ones leads to an additional document annotation useful for more robust retrieval.

2004

Using Networks to Visualize Publications

Dario Rodighiero

Retrieval systems are often shaped as lists organized in pages. However, the majority of users look at the first page ignoring the other ones. This presentation concerns an alterna- tive way to present the results of a query using network visualizations.  The presentation includes a case study that concerns a school of management. Its whole publications are arranged in a network visualization according to their lexical proximity, based on a technique called Term Frequency – Inverse Document Frequency (TF-IDF). These terms are further used to fill the space between the network nodes, creating a sort of semantic background. The case study shows pros and cons of such visual representa- tion through practical examples of term extraction and visualization interaction.