**Are you an EPFL student looking for a semester project?**

Work with us on data science and visualisation projects, and deploy your project as an app on top of GraphSearch.

Publication# An Algorithm Architecture Co-Design for CMOS Compressive High Dynamic Range Imaging

Antoine Dupret, William Guicquéro, Pierre Vandergheynst

*Ieee-Inst Electrical Electronics Engineers Inc, *2016

Journal paper

Journal paper

Abstract

Standard image sensors feature dynamic range about 60 to 70 dB while the light flux of natural scenes may be over 120 dB. Most imagers dedicated to address such dynamic ranges, need specific, and large pixels. However, canonical imagers can be used for high dynamic range (HDR) by performing multicapture acquisitions to compensate saturation. This technique is made possible at the expense of the need for large memory requirements and an increase of the overall acquisition time. On the other hand, the implementation of compressive sensing (CS) raises the same issues regarding the modifications of both the pixel and the read-out circuitry. Assuming HDR images are sufficiently sparse, CS claims they can be reconstructed from few random linear measurements. A novel CS-based image sensor design is presented in this paper allowing a compressive acquisition without changing the classical pixel design, as well as the overall sensor architecture. In addition to regular CS, HDR CS is enabled thanks to specific time diagrams of the control signals. An alternative nondestructive column-based readout mode constitutes the main change compared to a traditional functioning. The HDR reconstruction, which is also presented in this paper, is based on merging the information of multicapture compressed measurements while taking into account noise sources and nonlinearities introduced by both the proposed acquisition scheme and its practical implementation.

Official source

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Related concepts

Loading

Related publications

Loading

Related publications (6)

Loading

Loading

Loading

Related concepts (9)

Compressed sensing

Compressed sensing (also known as compressive sensing, compressive sampling, or sparse sampling) is a signal processing technique for efficiently acquiring and reconstructing a signal, by finding solu

Pixel

In digital imaging, a pixel (abbreviated px), pel, or picture element is the smallest addressable element in a raster image, or the smallest addressable element in a dot matrix display device. In m

Dynamic range

Dynamic range (abbreviated DR, DNR, or DYR) is the ratio between the largest and smallest values that a certain quantity can assume. It is often used in the context of signals, like sound and light. I

Over the past decade researches in applied mathematics, signal processing and communications have introduced compressive sampling (CS) as an alternative to the Shannon sampling theorem. The two key observations making CS theory widely applicable to numerous areas of signal processing are: i) due to their structural properties, natural signals typically have sparse representations in properly chosen orthogonal bases, ii) the number of linear non-adaptive measurements required to acquire high-dimensional data with CS is proportional to the signal’s sparsity level in the chosen basis. In multichannel signal applications the data of different channels is often highly correlated and therefore the unstructured sparsity hypothesis deployed by the classical CS theory results in suboptimal measurement rates. Meanwhile, the wide range of applications of multichannel signals and the extremely large and increasing flow of data in those applications motivates the development of more comprehensive models incorporating both inter and intra-channel data structures in order to achieve more efficient dimensionality reduction. The main focus of this thesis is on studying two new models for efficient multichannel signal compressed sensing. Our first approach proposes a simultaneous low-rank and joint-sparse matrix model for multichannel signals. As a result, we introduce a novel CS recovery scheme based on Nuclear-l2/l1 norm convex minimization for low-rank and joint-sparse matrix approximation. Our theoretical analysis indicates that using this approach can achieve significantly lower sampling rates for a robust multichannel data CS acquisition than state-of-the-art methods. More remarkably, our analysis confirms the near-optimality of this approach: the number of CS measurements are nearly proportional to the few degrees of freedom of such structured data. Our second approach introduces a stronger model for multichannel data synthesized by a linear mixture model. Here we assume that the mixture parameters are given as side information. As a result, multichannel data CS recovery turns into a compressive source separation problem, where we propose a novel decorrelating scheme to exploit the knowledge of mixture parameters for a robust and numerically efficient source identification. Our theoretical guarantees explain the fundamental limits of this approach in terms of the number of CS measurements, the sparsity level of sources, the sampling noise, and the conditioning of the mixture parameters. We apply these two approaches to compressive hyperspectral image recovery and source separation, and compare the efficiency of our methods to state-of-the-art approaches for several challenging real-world hyperspectral datasets. Note that applications of these methods are not limited to hyperspectral imagery and it can have a broad impact on numerous multichannel signal applications. As an example, for sensor network applications deploying compressive sampling schemes, our results indicate a tight tradeoff between the number of available sensors (channels) and the complexity/cost of each sensor. Finally, in a different multichannel signal application, we deal with a simple but very important problem in computer vision namely the detection and localization of people given their multi-view silhouettes captured by networks of cameras. The main challenge in many existing solutions is the tradeoff between robustness and numerical efficiency. We model this problem by a boolean (non-linear) inverse problem where, by penalizing the sparsity of the solution, we achieve accurate results comparable to state-of-the-art methods. More remarkably, using boolean arithmetics enables us to propose a real-time and memory efficient approximation algorithm that is mainly rooted in the classical literature of group testing and set cover.

New advances in the field of image sensors (especially in CMOS technology) tend to question the conventional methods used to acquire the image. Compressive Sensing (CS) plays a major role in this, especially to unclog the Analog to Digital Converters which are generally representing the bottleneck of this type of sensors. In addition, CS eliminates traditional compression processing stages that are performed by embedded digital signal processors dedicated to this purpose. The interest is twofold because it allows both to consistently reduce the amount of data to be converted but also to suppress digital processing performed out of the sensor chip. For the moment, regarding the use of CS in image sensors, the main route of exploration as well as the intended applications aims at reducing power consumption related to these components (i.e. ADC & DSP represent 99% of the total power consumption). More broadly, the paradigm of CS allows to question or at least to extend the Nyquist-Shannon sampling theory. This thesis shows developments in the field of image sensors demonstrating that is possible to consider alternative applications linked to CS. Indeed, advances are presented in the fields of hyperspectral imaging, super-resolution, high dynamic range, high speed and non-uniform sampling. In particular, three research axes have been deepened, aiming to design proper architectures and acquisition processes with their associated reconstruction techniques taking advantage of image sparse representations. How the on-chip implementation of Compressed Sensing can relax sensor constraints, improving the acquisition characteristics (speed, dynamic range, power consumption) ? How CS can be combined with simple analysis to provide useful image features for high level applications (adding semantic information) and improve the reconstructed image quality at a certain compression ratio ? Finally, how CS can improve physical limitations (i.e. spectral sensitivity and pixel pitch) of imaging systems without a major impact neither on the sensing strategy nor on the optical elements involved ? A CMOS image sensor has been developed and manufactured during this Ph.D. to validate concepts such as the High Dynamic Range - CS. A new design approach was employed resulting in innovative solutions for pixels addressing and conversion to perform specific acquisition in a compressed mode. On the other hand, the principle of adaptive CS combined with the non-uniform sampling has been developed. Possible implementations of this type of acquisition are proposed. Finally, preliminary works are exhibited on the use of Liquid Crystal Devices to allow hyperspectral imaging combined with spatial super-resolution. The conclusion of this study can be summarized as follows: CS must now be considered as a toolbox for defining more easily compromises between the different characteristics of the sensors: integration time, converters speed, dynamic range, resolution and digital processing resources. However, if CS relaxes some material constraints at the sensor level, it is possible that the collected data are difficult to interpret and process at the decoder side, involving massive computational resources compared to so-called conventional techniques. The application field is wide, implying that for a targeted application, an accurate characterization of the constraints concerning both the sensor (encoder), but also the decoder need to be defined.

The trends in the design of image sensors are to build sensors with low noise, high sensitivity, high dynamic range, and small pixel size. How can we benefit from pixels with small size and high sensitivity? In this dissertation, we study a new image sensor that is reminiscent of traditional photographic film. Each pixel in the sensor has a binary response, giving only a one-bit quantized measurement of the local light intensity. The response function of the image sensor is non-linear and similar to a logarithmic function, which makes the sensor suitable for high dynamic range imaging. We first formulate the oversampled binary sensing scheme as a parameter estimation problem based on quantized Poisson statistics. We show that, with a single-photon quantization threshold and large oversampling factors, the Cramér-Rao lower bound (CRLB) of the estimation variance approaches that of an ideal unquantized sensor, that is, as if there were no quantization in the sensor measurements. Furthermore, the CRLB is shown to be asymptotically achievable by the maximum likelihood estimator (MLE). By showing that the log-likelihood function is concave, we guarantee the global optimality of iterative algorithms in finding the MLE. We study the performance of the oversampled binary sensing scheme in presence of dark current noise. The noise model is an additive Bernoulli noise with a known parameter, and the noise only flips the binary output from "0" to "1". We show that the binary sensor is quite robust with respect to noise and its dynamic range is only slightly reduced. The binary sensor first benefits from the increasing of the oversampling factor and then suffers in term of dynamic range. We again use the MLE to estimate the light intensity. When the threshold is a single photon, we show that the log-likelihood function is still concave. Thus, the global optimality can be achieved. But for thresholds larger than "1", this property does not hold true. By proving that when the light intensity is piecewise-constant, the likelihood function is a strictly pseudoconcave function, we guarantee to find the optimal solution of the MLE using iterative algorithms for arbitrary thresholds. For the general linear light field model, the log-likelihood function is not even quasiconcave when thresholds are larger than "1". In this circumstance, we find an initial solution by approximating the light intensity field with a piecewise-constant model, and then we use Newton's method to refine the estimation using the exact model. We then examine one of the most important parameters in the binary sensor, i.e., the threshold used to generate binary values. We prove the intuitive result that large thresholds achieve better estimation performance for strong light intensities, while small thresholds work better for low light intensities. To make a binary sensor that works in a larger range of light intensities, we propose to design a threshold array containing multiple thresholds instead of a single threshold for the binary sensing. The criterion is to minimize the average CRLB which is a good approximation of the mean squared error (MSE). The performance analysis on the new binary sensor verifies the effectiveness of our design. Again, the MLE is used for reconstructing the light intensity field from the binary measurements. By showing that the log-likelihood function is concave for arbitrary threshold arrays, we ensure that the iterative algorithms can find the optimal solution of the MLE. Finally, we study the reconstruction problem for the binary image sensor under a generalized piecewise-constant light intensity field model, which is quite useful when parameters like oversampling factors are unknown. We directly estimate light exposure values, i.e., the number of photons hitting on each pixel. We assume that the light exposure values are piecewise-constant and we use the MLE for the reconstruction. This optimization problem is solved by iteratively working out two subproblems. The first one is to find the optimal light exposure value for each segment, given the optimal segmentation of the binary measurements. The second one is to find the optimal segmentation of the binary measurements given the optimal light exposure values for each segment. Several algorithms are provided for solving this optimization problem. Dynamic programming can obtain the optimal solution for 1-D signals, but the computation is quite heavy. To reduce the burden of computation, we propose a greedy algorithm and a method based on pruning of binary trees or quadtrees.