What is the Space of Spectral Sensitivity Functions for Digital Color Cameras?

Camera spectral sensitivity functions relate scene radiance with captured RGB triplets. They are important for many computer vision tasks that use color information, such as multispectral imaging, color rendering, and color constancy. In this paper, we aim to explore the space of spectral sensitivity functions for digital color cameras. After collecting a database of 28 cameras covering a variety of types, we find this space convex and two-dimensional. Based on this statistical model, we propose two methods to recover camera spectral sensitivities using regular reflective color targets (e.g., color checker) from a single image with and without knowing the illumination. We show the proposed model is more accurate and robust for estimating camera spectral sensitivities than other basis functions. We also show two applications for the recovery of camera spectral sensitivities — simulation of color rendering for cameras and computational color constancy.

Exploiting redundancy in color images

Joanna Marguier

The colors in a digital image do not only depend on the scene's reflectances, but also on the lighting conditions and on the characteristics of the imaging device. It is often necessary to separate the influence of these three elements, in particular to separate the contribution of the object's reflection properties from the one of the incident light. Both are usually represented by spectral quantities, i.e., defined over the visible spectrum. The retrieval of illuminant and reflectances from an RGB image, which provides three values per pixel, is an under-determined problem and requires additional information or assumption on the scene's content in order to be solved. In this thesis, we study how redundant information across a set of images can be exploited to recover either illuminant or reflectance knowledge. We demonstrate that accurate object color can be retrieved as tristimulus values from standard RGB images captured with an unknown camera under uncontrolled lighting conditions and no access to raw data, but containing a limited number of reference colors in the form of a specific calibration target. The correction must be constrained to a limited range of colors similar to the one of the object of interest, which are compared to pre-computed references for the derivation of a linear transform. We implement this method for two consumer oriented applications in Makeup and Home Décor, respectively, which provide users with color advice. The correction accuracy is of ΔE*ab ≃ 1 to 2 depending on the design of the target. We show that this accuracy is sufficient for the applications at hand. We also developed a method to retrieve illuminant spectra from a set of images taken with fixed location cameras, such as panoramic or surveillance ones. In pictures captured with such devices, there will be changes in lighting and dynamic content, but there will also be constant objects. We propose to use these elements as reference colors to solve for color constancy. More precisely, we exploit that their reflectances, while unknown, remain constant across images. We invert a series of image formation models in parallel for a set of test illuminants and, by forcing the output reflectances to match, deduce the illuminant under which each image was captured. Simulations on synthetic RGB patches demonstrate that real daylight illuminants can be retrieved with a median angular under 3° from sets containing as little as four images using a limited number of reference surfaces.

EPFL2010

Omnidirectional Light Field Analysis and Reconstruction

Luigi Bagnato

Digital photography exists since 1975, when Steven Sasson attempted to build the first digital camera. Since then the concept of digital camera did not evolve much: an optical lens concentrates light rays onto a focal plane where a planar photosensitive array transforms the light intensity into an electric signal. During the last decade a new way of conceiving digital photography emerged: a photography is the acquisition of the entire light ray field in a confined region of space. The main implication of this new concept is that a digital camera does not acquire a 2-D signal anymore, but a 5-D signal in general. Acquiring an image becomes more demanding in terms of memory and processing power; at the same time, it offers the users a new set of possibilities, like choosing dynamically the focal plane and the depth of field of the final digital photo. In this thesis we develop a complete mathematical framework to acquire and then reconstruct the omnidirectional light field around an observer. We also propose the design of a digital light field camera system, which is composed by several pinhole cameras distributed around a sphere. The choice is not casual, as we take inspiration from something already seen in nature: the compound eyes of common terrestrial and flying insects like the house fly. In the first part of the thesis we analyze the optimal sampling conditions that permit an efficient discrete representation of the continuous light field. In other words, we will give an answer to the question: how many cameras and what resolution are needed to have a good representation of the 4-D light field? Since we are dealing with an omnidirectional light field we use a spherical parametrization. The results of our analysis is that we need an irregular (i.e., not rectangular) sampling scheme to represent efficiently the light field. Then, to store the samples we use a graph structure, where each node represents a light ray and the edges encode the topology of the light field. When compared to other existing approaches our scheme has the favorable property of having a number of samples that scales smoothly for a given output resolution. The next step after the acquisition of the light field is to reconstruct a digital picture, which can be seen as a 2-D slice of the 4-D acquired light field. We interpret the reconstruction as a regularized inverse problem defined on the light field graph and obtain a solution based on a diffusion process. The proposed scheme has three main advantages when compared to the classic linear interpolation: it is robust to noise, it is computationally efficient and can be implemented in a distributed fashion. In the second part of the thesis we investigate the problem of extracting geometric information about the scene in the form of a depth map. We show that the depth information is encoded inside the light field derivatives and set up a TV-regularized inverse problem, which efficiently calculates a dense depth map of the scene while respecting the discontinuities at the boundaries of objects. The extracted depth map is used to remove visual and geometrical artifacts from the reconstruction when the light field is under-sampled. In other words, it can be used to help the reconstruction process in challenging situations. Furthermore, when the light field camera is moving temporally, we show how the depth map can be used to estimate the motion parameters between two consecutive acquisitions with a simple and effective algorithm, which does not require the computation nor the matching of features and performs only simple arithmetic operations directly in the pixel space. In the last part of the thesis, we introduce a novel omnidirectional light field camera that we call Panoptic. We obtain it by layering miniature CMOS imagers onto an hemispherical surface, which are then connected to a network of FPGAs. We show that the proposed mathematical framework is well suited to be embedded in hardware by demonstrating a real time reconstruction of an omnidirectional video stream at 25 frames per second.

EPFL2012

Aliasing mitigation in optical microscopy of dynamic biological samples by use of temporally modulated color illumination and a standard RGB camera

Christian Jaques, Michael Stefan Daniel Liebling

Significance: Despite recent developments in microscopy, temporal aliasing can arise when imaging dynamic samples. Modern sampling frameworks, such as generalized sampling, mitigate aliasing but require measurement of temporally overlapping and potentially negative-valued inner products. Conventional cameras cannot collect these directly as they operate sequentially and are only sensitive to light intensity. Aim: We aim to mitigate aliasing in microscopy of dynamic monochrome samples by implementing generalized sampling via the use of a color camera and modulated color illumination. Approach: We solve the overlap problem by spectrally multiplexing the acquisitions and using (positive) B-spline segments as projection kernels. Reconstruction involves spectral unmixing and inverse filtering. We implemented this method using a color LED illuminator. We evaluated its performance by imaging a rotating grid and its applicability by imaging the beating zebrafish embryo heart in transmission and light-sheet microscopes. Results: Compared to stroboscopic imaging, our method mitigates aliasing with performance improving as the projection order increases. The approach can be implemented in conventional microscopes but is limited by the number of available LED colors and camera channels. Conclusions: Generalized sampling can be implemented via color modulation in microscopy to mitigate temporal aliasing. The simple hardware requirements could make it applicable to other optical imaging modalities. (C) The Authors.