Analysis of Multiview Omnidirectional Images in a Spherical Framework

With the increasing demand of information for more immersive applications such as Google Street view or 3D movies, the efficient analysis of visual data from cameras has gained more importance. This visual information permits to extract some crucial information in scenes such as similarity for recognition, 3D scene information, textures and patterns of objects. Multi-camera systems provide detailed visual information about a scene with images from different positions and viewing angles. The conventional perspective cameras that are commonly used in these systems however, have limited field of view. Therefore, they require either the deployment of many of these cameras or the capture of many images from different points to extract sufficient details about a scene. It increases the amount of data to be processed and the maintenance costs for such systems. Omnidirectional vision systems overcome this problem due to their 360-degree field of view and have found wide application areas in robotics, surveillance. These systems often use special refractive elements such as fisheye lenses or mirror-lense systems. The resulting images however inherit from the specific geometry of these units. Therefore, the analysis of these images with methods designed for perspective cameras results in degraded performance in omnivision systems. In this thesis, we focus on the analysis of multi-view omnidirectional images for efficient scene information extraction. We propose a novel spherical framework for omnidirectional image processing by exploiting the property that most omnidirectional images can be uniquely mapped on the sphere. We propose solutions for three common multiview image processing problems, namely feature detection, dense depth estimation and super-resolution for omnidirectional images. We first address the feature extraction problem in omnidirectional images. We develop a scale-invariant feature detection method which carefully handles the geometry of the images by performing the scale-space analysis directly on their native manifolds such as the parabola or the sphere. We then propose a new descriptor and a matching criteria that take into account the geometry and also eliminate the need for orientation computation. We also demonstrate that the proposed method can be used to match features in images captured by different types of sensors such as perspective, omnidirectional or spherical cameras. We then propose a dense depth estimation method to extract the 3D scene information from multiple omnidirectional images. We propose a graph-cut method adapted to the geometry of those images to minimize an energy function formulated for the dense depth estimation problem. We also propose a parallel graph-cut method that gives a significant speed improvement without a big penalty in accuracy. We show that the proposed method can be applied to multi-camera depth estimation and depth-based arbitrary view synthesis. Finally, we consider the multi-view omnidirectional images with the transformation of pure rotation. We address the view synthesis problem for these images in the framework of super-resolution. Considering also the inaccuracies in the rotation parameters, we solve an optimization problem that jointly estimates the rotation errors and reconstructs a high resolution omnidirectional image. We then extend the minimization problem with a regularization term for improved reconstruction quality with a reduced number of images. Results with both synthetic and real omnidirectional images suggest that the proposed method is a viable solution for super-resolution with omnidirectional images. Overall, this dissertation addresses three important issues of multiview omnidirectional image analysis and processing in a novel spherical framework. Our feature detection method can be used for the calibration of omnidirectional images as well as the feature matching in mobile and hybrid camera networks. Furthermore, our dense depth estimation method can impact the quality of 3D scene reconstruction and provide efficient solutions for view synthesis and multiview omnidirectional image coding. Finally, our super-resolution algorithm for omnidirectional images can promote the development of efficient acquisition systems for high resolution omnidirectional images

Distributed Compressed Representation of Correlated Image Sets

Vijayaraghavan Thirumalai

Vision sensor networks and video cameras find widespread usage in several applications that rely on effective representation of scenes or analysis of 3D information. These systems usually acquire multiple images of the same 3D scene from different viewpoints or at different time instants. Therefore, these images are generally correlated through displacement of scene objects. Efficient compression techniques have to exploit this correlation in order to efficiently communicate the 3D scene information. Instead of joint encoding that requires communication between the cameras, in this thesis we concentrate on distributed representation, where the captured images are encoded independently, but decoded jointly to exploit the correlation between images. One of the most important and challenging tasks relies in estimation of the underlying correlation from the compressed correlated images for effective reconstruction or analysis in the joint decoder. This thesis focuses on developing efficient correlation estimation algorithms and joint representation of multiple correlated images captured by various sensing methodologies, e.g., planar, omnidirectional and compressive sensing (CS) sensors. The geometry of the 2D visual representation and the acquisition complexity vary for each sensor type. Therefore, we need to carefully consider the specific geometric nature of the captured images while developing distributed representation algorithms. In this thesis we propose robust algorithms in different scene analysis and reconstruction scenarios. We first concentrate on the distributed representation of omnidirectional images captured by catadioptric sensors. The omnidirectional images are captured from different viewpoints and encoded independently with a balanced rate distribution among the different cameras. They are mapped on the sphere which captures the plenoptic function in its radial form without Euclidean discrepancies. We propose a transform-based distributed coding algorithm, where the spherical images initially undergo a multi-resolution decomposition. The visual information is then split into two correlated partitions. The encoder transmits one partition after entropy coding, as well as the syndrome bits resulting from the Slepian-Wolf encoding of the other partition. The joint decoder estimates a disparity image to take benefit of the correlation between views and uses the syndrome bits to decode the missing information. Such a strategy proves to be beneficial with respect to the independent processing of images and shows only a small performance loss compared to the joint encoding of different views. The encoding complexity in the previous approach is non-negligible due to the visual information processing based on Slepian-Wolf coding and its associated rate parameter estimation. We therefore discard the Slepian-Wolf encoding and propose a distributed coding solution, where the correlated images are encoded independently using transform-based coding solutions (e.g., SPIHT). The central decoder now builds a correlation model from the compressed images, which is used to jointly decode a pair of images. Experimental results demonstrate that the proposed distributed coding solution improves the rate-distortion performance of the separate coding results for both planar and omnidirectional images. However, this improvement is significant only at medium to high bit rates. We therefore propose a rate allocation scheme that identifies and transmits the necessary visual information from each image to improve the correlation estimation accuracy at low bit rate. Experimental results show that for a given bit budget the proposed encoding scheme permits to compute an accurate correlation estimation comparing to the one obtained with SPIHT, JPEG 2000 or JPEG coding schemes. We show however that the improvement in the correlation estimation comes at the price of penalizing the image reconstruction quality; therefore there exists an interesting trade-off between the accurate correlation estimation and image reconstruction as encoding optimization objectives are different in both cases. Next, we further simplify the encoding complexity by replacing the classical imaging sensors with the simple CS sensors, that directly acquire the compressed images in the form of quantized linear measurements. We now concentrate on the particular problem, where one image is selected as the reference and it is used as a side information for the correlation estimation. We propose a geometry-based model to describe the correlation between the visual information in a pair of images. The joint decoder first captures the most prominent visual features in the reconstructed reference image using geometric functions. Since the images are correlated, these features are likely to be present in the other images too, possibly with geometric transformations. Hence, we propose to estimate the correlation model with a regularized optimization problem that locates these features in the compressed images. The regularization terms enforce smoothness of the transformation field, and consistency between the estimated images and the quantized measurements. Experimental results show that the proposed scheme is able to efficiently estimate the correlation between images for several multi-view and video datasets. The proposed scheme is finally shown to outperform DSC schemes based on unsupervised disparity (or motion) learning, as well as independent coding solutions based on JPEG 2000. We then extend the previous scenario to a symmetric decoding problem, where we are interested to estimate the correlation model directly from the quantized linear measurements without explicitly reconstructing the reference images. We first show that the motion field that represents the main source of correlation between images can be described as a linear operator. We further derive a linear relationship between the correlated measurements in the compressed domain. We then derive a regularized cost function to estimate the correlation model directly in the compressed domain using graph-based optimization algorithms. Experimental results show that the proposed scheme estimates an accurate correlation model among images in both multi-view and video imaging scenarios. We then propose a robust data fidelity term that improves the quality of the correlation estimation when the measurements are quantized. Finally, we show by experiments that the proposed compressed correlation estimation scheme is able to compete the solution of a scheme that estimates a correlation model from the reconstructed images without the complexity of image reconstruction. Finally, we study the benefit of using the correlation information while jointly reconstructing the images from the compressed linear measurements. We consider both the asymmetric and symmetric scenarios described previously. We propose joint reconstruction methodologies based on a constrained optimization problem which is solved using effective proximal splitting methods. The constraints included in our framework enforce the reconstructed images to satisfy both the correlation and the quantized measurements consistency objectives. Experimental results demonstrate that the proposed joint reconstruction scheme improves the quality of the decoded images, when compared to a scheme where the images are handled independently. In this thesis we build efficient distributed scene representation algorithms for the multiple correlated images captured in planar, omnidirectional and CS cameras. The coding rate in our symmetric distributed coding solution stays balanced between the encoders and stays close to the joint encoding solutions. Our novel algorithms lead to effective correlation estimation in different sensing and coding scenarios. In addition, we provide innovative solutions for robust correlation estimation from highly compressed images in simple sensing frameworks. Our CS-based joint reconstruction frameworks effectively exploit the inter-view correlation, that permits to achieve high compression gains compared to state-of-the-art independent and distributed coding solutions.

EPFL2012

Exploitation de connaissances sémantiques externes dans les représentations vectorielles en recherche documentaire

Florian Seydoux

The work presented in this thesis deals with several problems met in information retrieval (IR), task which one can summarise as identifying, in a collection of "documents", a subset of documents carrying a sought information, i.e.. relevant for a request expressed by a user. In the case of textual documents, to which we limited ourselves within the framework of this thesis, a significant part of the difficulty lies in ambiguity inherent to human languages. The interaction with the user is also approached in our work, by studying a tool enabling a natural language access to a database. Finally, some techniques which permit the visualisation of large collections of documents are also presented. In this document we first of all describe the principal models of IR by highlighting the relations which exist with some manual technics of IR and document retrieval, developed during the past centuries. We present the principle of document indexing, allowing us to represent documents in a multidimensional space, and the use of this representation by a vectorial model. After having reviewed the principal improvements made these last years with vectorial research systems, including the preprocessings of collections, the indexing mechanism and measurements of similarities between documents, we detail some recent usecases of additional semantic resources (semantic dictionaries, thesaurus, networks, ontologies) reported in scientific literature for the indexing task. We then present more in detail the semantic indexing principle of textual documents by using a thesaurus, consisting in integrating in the document's representation space at least part of the informational contents of hierarchical semantic resources. We propose a general framework allowing us to describe and position various possible techniques to carry out the semantic indexing by adapting, if possible, the specificity of the descriptions resulting from the semantic resources to the data to be represented. We use this framework to describe three families of criteria usable for semantic indexing, each one having its own characteristics. For each of these families, we give the specific algorithms allowing the computation of the criteria. The first two families allow us to consider several criteria already known in feature selection. Moreover we show that, unfortunately, many of these criteria are in fact not very effective for the considered task. The third family allows us to introduce a completely new criterion, the Minimum Redundancy Cut criterion (MRC), built on the basis of the information theory and allowing us to obtain index terms having a probability of occurrence in the collection of documents as well balanced as possible. Finally, we treat the case of semantic index independent of the data (statically choosen), allowing a parameterisation of the level of generality of the index terms. Some of the criteria suggested for semantic indexing has been empirically evaluated. To judge their relevance, we used a well known vectorial system (the Smart IR system) and measured the performances of IR obtained with various reference collections. Those collections was indexed on the basis of the studied criterion, by taking into account the strongly structuring semantic relation of hyper/hyponymy ("is-a" relation), given by two different semantic resources. By comparing results obtained with the performances of a traditional indexing (using the lemmas of the words as representation space), we can show on one hand the relevance of the semantic indexings (in RD) and on the other hand the quality of the proposed criterion (MRC). Concerning man-machine interaction, we present a general outline allowing to build in a relatively fast and systematic way systems with mixed initiative, giving the human user a large (and natural) latitude in the control of the dialogue. This outline is usable in typical database research-task applications (where the database is hidden to the user, but the latter knows exactly which information they wish to find) as well as advice-task applications, for which the users does not necessarily have a precise idea of their needs, and uses the system not only for specifing their wishes, but also a set of propositions as a final result. We particularly stress the techniques allowing us to obtain a robust system, able to deal with speech recognizer failures. Concerning the visualisation of large textual data collections, we present an application of the correspondences analysis (allowing to highlight similarities and oppositions for various groups of entity, built on the basis of additional features present in the DB) to the case of patents data. In addition, we propose a method (based on the bootstrap replication principle) allowing us to determine a confidence interval for relative positionings of various groups, thus permit to immediately judge the reliability of the visually apparent similarities or oppositions.

EPFL2006

Scale Invariant Features and Polar Descriptors in Omnidirectional Imaging

Zafer Arican, Pascal Frossard

We propose a method to compute scale invariant features in omnidirectional images. We present a formulation based on Riemannian geometry for the definition of differential operators on non-Euclidian manifolds that describe the mirror and lens structure in omnidirectional imaging. These operators lead to a scale-space analysis that preserves the geometry of the visual information in omnidirectional images. We then build a novel scale-invariant feature detection framework for any omnidirectional image that can be mapped on the sphere. We also present a new descriptor and feature matching solution for omnidirectional images. The descriptor builds on the log-polar planar descriptors and adapts the descriptor computation to the specific geometry and the non-uniform sampling density of spherical and omnidirectional images. We further propose a rotation-invariant matching method that eliminates the orientation computation during the feature detection phase and thus decreases the computational complexity. Finally, we show that the proposed framework also permits to match features in images with different geometries. Experimental results demonstrate that the new feature detection method combined with the proposed descriptors offers promising performance and improves on the common SIFT features computed on the planar omnidirectional images as well as other state-of-the art methods for omnidirectional images.