Image-Based Acquisition and Modeling of Polarimetric Reflectance

Realistic modeling of the bidirectional reflectance distribution function (BRDF) of scene objects is a vital prerequisite for any type of physically based rendering. In the last decades, the availability of databases containing real-world material measurements has fueled considerable innovation in the development of such models. However, previous work in this area was mainly focused on increasing the visual realism of images, and hence ignored the effect of scattering on the polarization state of light, which is normally imperceptible to the human eye. Existing databases thus only capture scattered flux, or polarimetric BRDF datasets are too directionally sparse (e.g., in-plane) to be usable for simulation. While subtle to human observers, polarization is easily perceived by any optical sensor (e.g., using polarizing filters), providing a wealth of additional information about shape and material properties of the object under observation. Given the increasing application of rendering in the solution of inverse problems via analysis-by-synthesis and differentiation, the ability to realistically model polarized radiative transport is thus highly desirable. Polarization depends on the wavelength of the spectrum, and thus we provide the first polarimetric BRDF (pBRDF) dataset that captures the polarimetric properties of real-world materials over the full angular domain, and at multiple wavelengths. Acquisition of such reflectance data is challenging due to the extremely large space of angular, spectral, and polarimetric configurations that must be observed, and we propose a scheme combining image-based acquisition with spectroscopic ellipsometry to perform measurements in a realistic amount of time. This process yields raw Mueller matrices, which we subsequently transform into Rusinkiewicz-parameterized pBRDFs that can be used for rendering. Our dataset provides 25 isotropic pBRDFs spanning a wide range of appearances: diffuse/specular, metallic/dielectric, rough/smooth, and different color albedos, captured in five wavelength ranges covering the visible spectrum. We demonstrate usage of our data-driven pBRDF model in a physically based renderer that accounts for polarized interreflection, and we investigate the relationship of polarization and material appearance, providing insights into the behavior of characteristic real-world pBRDFs.

Joint Acquisition of Color and Near-Infrared Images on a Single Sensor

Zahra Sadeghipoor Kermani

In the past few years, fusing NIR and color images has been explored in general computational photography and computer vision tasks, where traditionally only color images are used. The additional information provided by the differences of light and scene reflections in the visible and NIR bands of the electromagnetic spectrum is used in several applications such as image denoising, image dehazing, shadow detection and removal, and high dynamic-range imaging. In this thesis, we study a system that simultaneously captures color and NIR images on a single silicon sensor. Such a camera could be manufactured with minor changes in the hardware of consumer color cameras and it could be integrated inside small devices such as cell phones. We address two main challenges in color and NIR acquisition. First, we study the spatial and spectral sampling of the scene, which is inevitable in single-sensor acquisition of multiple spectral channels. We then focus on chromatic aberration distortions. Similarly to color imaging, we use a color filter array (CFA) to sample the scene in visible and NIR bands. We address two main challenges regarding the CFA: (1) designing the CFA, and (2) developing a demosaicing algorithm to reconstruct full-resolution images from the subsampled measurements. We consider a general CFA with filters that transmit different mixtures of color and NIR channels. We develop a framework that, by exploiting the spatial and spectral correlations of color and NIR images, computes the transmittance of each filter and the demosaicing matrix. Our optimized CFA and demosaicing outperform other solutions developed for single-sensor color and NIR acquisition. We also investigate a CFA that is formed by one blue, one green, one red, and one NIR-pass filter. We call it the RGBN CFA and assume that, similarly to color cameras, it uses dye filters that do not have sharp cut-offs. Hence, color and NIR radiations leak into NIR and color filters, respectively. We devise an algorithm that reconstructs full-resolution images from mixed and subsampled sensor measurements. The RGBN CFA and our reconstruction algorithm perform as well as or even better than other single-sensor acquisition techniques that use more complicated hardware components. The problem of chromatic aberration is caused by deficiencies of optical elements. A simple lens converges light rays with different wavelengths at different distances from the lens. Hence, if the color image is in focus and sharp on the sensor plane, the NIR image captured with the same focus settings is out of focus and blurred. We propose an algorithm that retrieves the lost details in NIR using the gradients of the sharp color image. As the high-frequency details of color and NIR images are not strongly correlated in all image patches, our method locally adapts the contribution of color gradients in deblurring. To achieve this, we develop a multiscale scheme that iterates between deblurring NIR and estimating the correlation between color and NIR high-frequency components. Our algorithm outperforms both blind and guided deblurring approaches. We also design a method that estimates a dense blur-kernel map when the severity of chromatic aberration changes as the depths of objects vary across the image. Our method performs better than the competing methods both in estimating the blur-kernel map and in deblurring.

EPFL2015

Data-driven Measurement Designs for Magnetic Resonance Imaging

Baran Gözcü

Despite being a powerful medical imaging technique which does not emit any ionizing radiation, magnetic resonance imaging (MRI) always had the major problem of long scanning times that can take up to an hour depending on the application. It also requires uncomfortable breath-holds due to the slow acquisition, sedation of children and repeated scans in the cases of degraded image quality due to body motion. Recent years have seen new image reconstruction techniques that need less amount of acquired data (i.e., accelerated scans) for reconstructing MRI images: parallel imaging and compressed sensing (CS). Although much work has been done on the reconstruction side, there has been relatively less work on experimental design, i.e., which parts of the Fourier domain to acquire during the scan, an essential factor that considerably affects the performance of image reconstructions. The state-of-the-art experimental designs use random subsampling either based on parametric models or heuristical adaptive models. The requirement of extensive parameter tuning and the random nature of the performance render these methods impractical and unreliable for clinical use. Can we systematically use the data acquired during past MRI scans for the design of accelerated scans with a reliable image quality? This problem is the focus of this thesis which proposes a data-driven scan design approach and training procedures which efficiently and effectively learn the structure inherent in the data, and accordingly, design the scans that directly acquire only the most relevant information during acquisition given an acceleration rate constraint. As a result, this boosts the performance of the existing state-of-the-art compressive sensing techniques on real-world datasets. Moreover, this approach provides strong theoretical guarantees by using tools from statistical learning theory. The intensive computational training procedures of our approach are made feasible by large-scale implementations on a parallel computing cluster. In return, this approach avoids any dependence on parametric or heuristic models and provides a reliably consistent image reconstruction performance for accelerated scans. Our approach is flexible and capable of giving deterministic scan designs specific to the anatomy, to the acceleration rate in use, to the reconstruction algorithm and the scan settings such as static/dynamic and parallel imaging. Apart from measurement designs for MRI, this thesis also considers the reconstruction problem. In particular, we focus on the inverse problems that involve a mixture of regularizers in the objective function, exploiting multiple structures at the same time. For these problems, we propose a reliable and systematic optimization framework and illustrate its effectiveness. Finally, in the last part of the thesis, we present a data-driven model and an optimization method for the design of nearly isometric, linear and dimensionality reducing embeddings.

EPFL2019

Optical spectral probing for epithelial tissue characterisation

Philippe Thueler

In this thesis, a new method has been developed, which determines the optical coefficients spectra (absorption and scattering) of small volumes of tissue (1 mm3) by the measurement of spatially resolved reflectance between 480 and 950 nm. Firstly, an algorithm has been developed to determine the optical coefficients form reflectance measurements: It is based on the optical propagation model introduced by Bevilacqua [Bevilacqua 99 JOSA]. This model takes the first two moments of the phase function into account through the γ parameter, allowing to accurately simulate the propagation of photons close to the source by the Monte Carlo method. These simulations are then fitted to the reflectance to obtain the optical coefficients. To obtain any value of absorption, scattering or γ parameter, the simulations are interpolated by 3D B-splines. A complete robustness to noise study of the algorithm has been presented. A theoretical study of the γ parameter in tissue over broad wavelengths has been reported. Tissue was simulated with Mie theory for a fractal distribution of sphere diameters. For such a distribution, the γ spectrum is almost flat, instead of being wavy like in the case of mono-disperse sphere diameter. The correlation of the mean value of γ with the fractal power (or dimension) of the distribution was made and showed an analytical form of hyperbolic tangent. It allowed deepening the interpretation of the γ parameter as being an indicator of the relative number of small scatterers (much smaller than the wavelength, e.g. of Rayleigh type) in comparison of bigger ones (their size being comparable to the wavelength). The assessment of the set-up and the algorithm was made by measuring the reflectance of a solution containing 5 different diameters of polystyrene spheres simulating the scattering and Nigrosin colorant as absorber. It showed a good agreement between measurements and prediction. In order to built a better comprehension of absorption and scattering at the tissue scale, a first part of in vivo investigations has been made on an animal model. The skin of the mouse back was altered with Freund's adjuvant and TPA and the optical coefficients were measured on treated and control sites. The effect of alteration (epidermis and dermis thickening, oil vesicle appearing, inflammatory cells infiltration, etc.) was clearly correlated with optical coefficients variations. The scattering spectrum always showed the better contrast. The absorption spectrum, especially in the haemoglobin peak region (between 550 and 600 nm) also showed significant information, in terms of oxy and deoxy-haemoglobin concentration and saturation. Sensitivity and specificity of the "diagnosis" was evaluated. The study of stomach epithelium in vivo on 29 human subjects was made using a small optical fibre probe (

EPFL2001