Demultiplexing Visible and Near-Infrared Information in Single- Sensor Multispectral Imaging

In this paper, we study a single-sensor imaging system that uses a multispectral filter array to spectrally sample the scene. Our system captures information in both visible and near-infrared bands of the electromagnetic spectrum. Due to manufacturing limitations, the visible filters in this system also transmit the NIR radiation. Similarly, visible light is transmitted by the NIR filter, leading to inaccurate mixed spectral measurements. We present an algorithm that resolves this issue by separating NIR and visible information. Our method achieves this goal by exploiting the correlation of multispectral images in both spatial and spectral domains. Simulation results show that the mean square error of the data corrected by our method is less than 1/20 of the error in sensor spectral measurements.

Joint Acquisition of Color and Near-Infrared Images on a Single Sensor

Zahra Sadeghipoor Kermani

In the past few years, fusing NIR and color images has been explored in general computational photography and computer vision tasks, where traditionally only color images are used. The additional information provided by the differences of light and scene reflections in the visible and NIR bands of the electromagnetic spectrum is used in several applications such as image denoising, image dehazing, shadow detection and removal, and high dynamic-range imaging. In this thesis, we study a system that simultaneously captures color and NIR images on a single silicon sensor. Such a camera could be manufactured with minor changes in the hardware of consumer color cameras and it could be integrated inside small devices such as cell phones. We address two main challenges in color and NIR acquisition. First, we study the spatial and spectral sampling of the scene, which is inevitable in single-sensor acquisition of multiple spectral channels. We then focus on chromatic aberration distortions. Similarly to color imaging, we use a color filter array (CFA) to sample the scene in visible and NIR bands. We address two main challenges regarding the CFA: (1) designing the CFA, and (2) developing a demosaicing algorithm to reconstruct full-resolution images from the subsampled measurements. We consider a general CFA with filters that transmit different mixtures of color and NIR channels. We develop a framework that, by exploiting the spatial and spectral correlations of color and NIR images, computes the transmittance of each filter and the demosaicing matrix. Our optimized CFA and demosaicing outperform other solutions developed for single-sensor color and NIR acquisition. We also investigate a CFA that is formed by one blue, one green, one red, and one NIR-pass filter. We call it the RGBN CFA and assume that, similarly to color cameras, it uses dye filters that do not have sharp cut-offs. Hence, color and NIR radiations leak into NIR and color filters, respectively. We devise an algorithm that reconstructs full-resolution images from mixed and subsampled sensor measurements. The RGBN CFA and our reconstruction algorithm perform as well as or even better than other single-sensor acquisition techniques that use more complicated hardware components. The problem of chromatic aberration is caused by deficiencies of optical elements. A simple lens converges light rays with different wavelengths at different distances from the lens. Hence, if the color image is in focus and sharp on the sensor plane, the NIR image captured with the same focus settings is out of focus and blurred. We propose an algorithm that retrieves the lost details in NIR using the gradients of the sharp color image. As the high-frequency details of color and NIR images are not strongly correlated in all image patches, our method locally adapts the contribution of color gradients in deblurring. To achieve this, we develop a multiscale scheme that iterates between deblurring NIR and estimating the correlation between color and NIR high-frequency components. Our algorithm outperforms both blind and guided deblurring approaches. We also design a method that estimates a dense blur-kernel map when the severity of chromatic aberration changes as the depths of objects vary across the image. Our method performs better than the competing methods both in estimating the blur-kernel map and in deblurring.

EPFL2015

Mid-infrared surface plasmon resonance sensing applied to refractive index measurements of liquids

Sylvain Herminjard

There has been a very strong development of the sensors based on surface plasmon resonance during the last thirty years, mostly for biological and biomedical applications. If the first experiments in this field were carried out at the beginning of the 20t century, it would have been necessary to wait for the end of the sixties to see the first devices performing accurate measurements of the refractive indices of gas and liquid samples. Indeed, the surface plasmon resonance phenomenon allows to measure refractive index variations with a high sensitivity that is hardly obtained with other optical measurement techniques. The basic principle consists of exciting the free electrons of a metallic layer with light, which induces the formation of a surface wave called surface plasmon. This resonant phenomenon depends on the wavelength of the excitation light and the refractive indices of the metallic layer and the material located close to the metallic layer. A variation of the refractive index of the latter induces a variation of the resonance condition, which allows to realize optical sensors designed to measure determined samples. An appropriate calibration allows to set a relationship between the variation of the concentration of the sample and the measured refractive index value. Among the various parameters that describe the performances of the sensor, one of the most important is the sensitivity. It is possible to demonstrate that the excitation wavelength has an impact on the sensitivity of the system : the latter increases if the measured sample has an absorption peak at a wavelength close to the excitation wavelength. The main goal of this thesis is to explore this sensitivity enhancement technique based on the absorption properties of the probed medium. A proof-of-principle is experimentally demonstrated with an optical setup constituted of laser sources working in the visible range of the electromagnetic spectrum. However, many materials have absorption peaks located in the mid-infrared range due to their fundamental molecular vibrations. Therefore, the realization of a plasmonic sensor working in the mid-infrared could allow to measure these materials with an increased sensitivity. To the best of our knowledge, surface plasmon resonance based sensors have never been developed above the near-infrared. This thesis presents for the first time the design and realization of a plasmonic sensor working in the mid-infrared range able to measure refractive index variations of gas and liquids samples. Even if the optical sources, components and detectors designed for the mid-infrared are clearly different from the ones designed for the visible range, it has been possible to successfully realize two systems working in the mid-infrared range during this project. The first is able to perform gas measurements. As the excitation wavelength depends on the measured sample, it was necessary to select a laser source that had a wavelength located close to the absorption peak of the gas under measurement. For this purpose, we designed a plasmonic sensor working at a 4.4 µm wavelength able to measure carbon dioxide (CO2) absorbing at 4.3 µm. It has been demonstrated that the experimental sensitivity of this system is five times higher than the theoretical sensitivity obtained with a system working in the visible range at a 633 nm wavelength. The second has been conceived to measure liquid samples in order to measure variations of concentration of CO2 dissolved in water. If measurements of refractive index variations (without using the sensitivity enhancement technique) could be performed successfully with variations of concentration of alcohol in water, further work is necessary to get decisive results for CO2 dissolved in water. This project was carried out in collaboration with an industrial partner in order to take into account some industrial constraints during the development of the sensor. Therefore, particular care has been taken in order to design a system that is compact, mechanically robust, without any mechanical moving parts, cheap and easily reproducible. The technological development presented in this thesis finds many applications mainly in the chemical, biological and biomedical fields. Indeed, many proteins show absorption peaks in the mid-infrared. It could be possible to measure their concentration or the modification of their properties – like their composition or structure – with an increased sensitivity. The interest of surface plasmons is that the measurement is not a transmission-based technique, but rather a "reflection-based technique", as the light does not go through the gas or liquid sample. This measurement technique is also adapted for high volume samples that are present in industrial production processes like water or soft drink quality control.

EPFL2010

Geometry-aware analysis of high-dimensional visual information sets

Effrosyni Kokiopoulou

Over the past few decades we have been experiencing a data explosion; massive amounts of data are increasingly collected and multimedia databases, such as YouTube and Flickr, are rapidly expanding. At the same time rapid technological advancements in mobile devices and vision sensors have led to the emergence of novel multimedia mining architectures. These produce even more multimedia data, which are possibly captured under geometric transformations and need to be efficiently stored and analyzed. It is also common in such systems that data are collected distributively. This very fact poses great challenges in the design of effective methods for analysis and knowledge discovery from multimedia data. In this thesis, we study various instances of the problem of classification of visual data under the viewpoint of modern challenges. Roughly speaking, classification corresponds to the problem of categorizing an observed object to a particular class (or category), based on previously seen examples. We address important issues related to classification, namely flexible data representation for joint coding and classification, robust classification in the case of large geometric transformations and classification with multiple object observations in both centralized and distributed settings. We start by identifying the need for flexible data representation methods that are efficient in both storage and classification of multimedia data. Such flexible schemes offer the potential to significantly impact the efficiency of current multimedia mining systems, as they permit the classification of multimedia patterns directly in their compressed form, without the need for decompression. We propose a framework, called semantic approximation, which is based on sparse data representations. It formulates dimensionality reduction as a matrix factorization problem, under hybrid criteria that are posed as a trade-off between approximation for efficient compression and discrimination for effective classification. We demonstrate that the proposed methodology competes with state-of-the-art solutions in image classification and face recognition, implying that compression and classification can be performed jointly without performance penalties with respect to expensive disjoint solutions. Next, we allow the multimedia patterns to be geometrically transformed and we focus on transformation invariance issues in pattern classification. When a pattern is transformed, it spans a manifold in a high dimensional space. We focus on the problem of computing the distance of a certain test pattern from the manifold, which is also closely related to the image alignment problem. This is a hard non-convex problem that has only been sub-optimally addressed before. We represent transformation manifolds based on sparse geometric expansions, which results in a closed-form representation of the manifold equation with respect to the transformation parameters. When the transformation consists of a synthesis of translations, rotations and scalings, we prove that the objective function of this problem can be decomposed as a difference of convex functions (DC). This very property allows us to solve optimally our optimization problem with a cutting plane algorithm, which is well known to successfully find the global minimizer in practice. We showcase applications in robust face recognition and image alignment. The classification problem with multiple observations is addressed next. Multiple observations are typically produced in practice when an object is observed over successive time instants or under different viewpoints. In particular, we focus on the problem of classifying an object when multiple geometrically transformed observations of it are available. These multiple observations typically belong to a manifold and the classification problem resides in determining which manifold the observations belong to. We show that this problem can be viewed as a special case of semi-supervised learning, where all unlabelled examples belong to the same class. We design a graph-based algorithm, which exploits the structure of the manifold. Estimating the unknown object class then results into a discrete optimization problem that can be solved efficiently. We show the performance of our algorithm in classification of multiple handwritten digit images and in video-based face recognition. Next, we study the problem of classification of multiple observations in distributed scenarios, such as camera networks. In this case the classification is performed iteratively and distributively, without the presence of a central coordinator node. The main goal is to reach a global classification decision using only local computation and communication, white ensuring robustness to changes in network topology. We propose to use consensus algorithms in order to design a distributed version of the aforementioned graph-based algorithm. We show that the distributed classification algorithm has similar performance as its centralized counterpart, provided that the training set is sufficiently large. Finally, we delve further into the convergence properties of consensus-based distributed algorithms and we propose an acceleration methodology for fast convergence that uses the memory of the sensors. Our simulations show that the convergence is indeed accelerated in both static and dynamic networks, and that distributed classification in sensor networks can significantly benefit from them. Overall, the present thesis addresses a few important issues related to pattern analysis and classification in modern multimedia systems. Our solutions for semantic approximation and transformation invariance can impact the efficiency and robustness of classification in multimedia systems. Furthermore, our graph-based framework for multiple observations is able to perform effective classification in both centralized and distributed environments. Finally, our fast consensus algorithms can significantly contribute to the accelerated convergence of distributed classification algorithms in sensor networks.

EPFL2009