Active illumination and computational Methods for Temporal and Spectral Super-Resolution Microscopy

Light microscopy is a tool of paramount importance for biologists and has been constantly improved for the past four centuries. Despite many recent developments, microscopy techniques still require improvement, especially to reach better temporal and spectral resolutions. In particular, many high-end microscopes favor mostly spatial resolution, at the expense of the latter two types of resolution. In this thesis, we present methods based on the use of active illumination and computational algorithms to increase temporal and spectral resolutions of microscopes. Our methods aim to provide users with the flexibility to chose, within a single instrument, which type of resolution is to be favored based on the application at hand. More generally, our approach has fundamental implications on the signal sensing procedure, allowing, for example, to mitigate temporal aliasing in sequences of images.

Our first method performs temporal super-resolution imaging of monochrome scenes using a hue-encoded shutter. By making use of an active multi-spectral illumination, temporal information is encoded in the hue of the acquisitions. We characterize the method showing a resolution improvement of 2.8 and an increase of frame-rate of a factor 3. We demonstrate the applicability of our method to bright-field transmission microscopy by applying the method to the beating heart of a zebrafish.We then extend this method to fluorescence microscopy. We add a temporal regularization term to make the method robust to fluorescent labelings inhomogeneities. We present an application of the method to the beating heart of a zebrafish that emits fluorescent light of two different colors. Implementing our method within a light-sheet microscope allows us to reconstruct 3D+time videos of the beating heart at twice the acquisition frame-rate.

Our second method offers a way to perform temporal generalized sampling by computing simultaneous inner products with the sampled signal. Similarly to the first method, we take advantage of working with multiple illumination hues to compute as many simultaneous inner products, which we retrieve via an unmixing procedure. We use equivalent basic and dual B-splines representations to ensure having finite-length and positive pre-filters, as well as finite-support reconstruction functions. We show applications of our method to a fast rotating target, as well as to the beating heart of a zebrafish, both in transmission and fluorescence microscopy.

Finally, we introduce a method to perform spectral imaging of repeating processes, such as the beating heart. The method sequentially acquires multiple movies with various filters, performs temporal registration of all movies and reconstructs a spectral movie through solving of a spectral unmixing problem, pixel by pixel, at each time point. We characterize the method and show a median error of approximately 10%, by comparing reconstructions on a static sample from our method with measurements obtained with a spectrometer. We then perform validation by comparing static reconstructions with dynamic ones of the same sample. We demonstrate the potential of the method to microscopy by performing spectral imaging of the beating heart of a zebrafish.

Taken together, these methods offer a versatile toolbox to improve the temporal or spectral resolution in both bright field and fluorescence microscopy, which we foresee could be directly implemented in a number of specialized instruments.

Compact lensless digital holography

Manon Rostykus

Microscopy is of high interest for biology since it allows imaging features that are too small to be seen with naked eyes. However, cells are mostly transparent to visible and infrared light which makes it difficult to see with a traditional microscope. To do so, phase imaging was invented by F. Zernike, who received the Nobel prize in 1953 for this invention. It provides intensity contrast to visualize transparent samples such as those found in biology without any staining. Among the different implementations of phase imaging, digital holographic microscopy (DHM) is a well-known phase method which provides a quantitative value of the phase generated by the sample under test. Implementations of DHMs without the help of any lens element (so called lensless) offer the added advantage to provide large field of views (severalmm2 compared to several hundred ¹m2) and more compact setups than traditional DHMs which have high quality microscope objectives. The lateral resolution is limited by the pixel size of the camera. However, several techniques have been developed to obtain subpixel resolution with lensless setups, hence giving a resolution smaller than the pixel size. In this thesis, two lensless transmission DHMs are presented using a side illumination technique in order to further reduce the device size and keep a visual access to the sample. The first one operates in an in-line configuration: the most compact but using time-consuming image post-processing. The second one is also lensless and uses an off-axis configuration allowing quasi-real time imaging but less compact. Their practical use is described and images of transparent (phase only) samples are shown. Super-resolution (subpixel) using several subpixel shifted holograms was then investigated for the in-line configuration and demonstrated on absorptive samples. Since those microscopes are compact and made out of off-the-shelf low priced elements, they could be broadly spread in schools for educational purposes. They could also be implemented in cells incubators, allowing visualization inside processes at low cost and multiplication of the number of measurements made at the same time.

EPFL2018

Time-Varying Segmentation for Mapping of Land Cover Changes

François Aspert, Meritxell Bach Cuadra, Jean-Philippe Thiran

Among all satellite imaging systems, Synthetic Aperture Radar (SAR) is useful for monitoring purposes, as they provide data at any time under all weather conditions. Today, three spaceborne SAR systems are operational, while by the end of 2007 three further SAR instruments will be launched, thereby allowing to obtain an almost continuous monitoring of the Earth coverage. In order to translate these multi-temporal data into information in an unsupervised (automated) and reliable way, sophisticated algorithms must be available. In the past years several approaches - primarily based on texture analysis and statistical scene estimation - have been proposed for multi-temporal SAR data analysis. Such methods, based on probability density functions, perform well under strictly controlled conditions, but they are often limited with respect to sensor synergy where complex joint probability density functions must be considered - and to the temporal aspect. To address these limitations, an original set of algorithms for the unsupervised multi-temporal SAR data analysis is proposed. To this end, several issues have been tackled: filtering, edge detection, and image segmentation. Moreover, condition sine qua non for the system design, was that i) it is sensor independent, and ii) data from different SAR systems can be ingested without any a-priori knowledge about the probability density function. Basically, the proposed time varying segmentation involves four independent steps. In a first step, a multi-temporal anisotropic non-linear diffusion filter is applied to filter the images which ultimately allow feature extraction. Subsequently, an extension of Canny edge detection algorithm for multi-temporal edge detection is applied, hence obtaining an edge map consistent across the whole sequence of temporal images. In a third step, closed regions are obtained using a two-part coding scheme with the edge map (as side information) and region growing technique (using the multi-temporal stack). Finally, the number of underlying spectral class composing a segment histogram at every frame is estimated, thus detecting changes due to temporal land cover fluctuations. Results are presented based on a set of 16 ENVISAT ASAR Alternating Polarization and Radarsat-1 Fine Beam images acquired between November 2004 and July 2005 for an agricultural (maize and sun flower) area in South Africa.

2007

Distributed Compressed Representation of Correlated Image Sets

Vijayaraghavan Thirumalai

Vision sensor networks and video cameras find widespread usage in several applications that rely on effective representation of scenes or analysis of 3D information. These systems usually acquire multiple images of the same 3D scene from different viewpoints or at different time instants. Therefore, these images are generally correlated through displacement of scene objects. Efficient compression techniques have to exploit this correlation in order to efficiently communicate the 3D scene information. Instead of joint encoding that requires communication between the cameras, in this thesis we concentrate on distributed representation, where the captured images are encoded independently, but decoded jointly to exploit the correlation between images. One of the most important and challenging tasks relies in estimation of the underlying correlation from the compressed correlated images for effective reconstruction or analysis in the joint decoder. This thesis focuses on developing efficient correlation estimation algorithms and joint representation of multiple correlated images captured by various sensing methodologies, e.g., planar, omnidirectional and compressive sensing (CS) sensors. The geometry of the 2D visual representation and the acquisition complexity vary for each sensor type. Therefore, we need to carefully consider the specific geometric nature of the captured images while developing distributed representation algorithms. In this thesis we propose robust algorithms in different scene analysis and reconstruction scenarios. We first concentrate on the distributed representation of omnidirectional images captured by catadioptric sensors. The omnidirectional images are captured from different viewpoints and encoded independently with a balanced rate distribution among the different cameras. They are mapped on the sphere which captures the plenoptic function in its radial form without Euclidean discrepancies. We propose a transform-based distributed coding algorithm, where the spherical images initially undergo a multi-resolution decomposition. The visual information is then split into two correlated partitions. The encoder transmits one partition after entropy coding, as well as the syndrome bits resulting from the Slepian-Wolf encoding of the other partition. The joint decoder estimates a disparity image to take benefit of the correlation between views and uses the syndrome bits to decode the missing information. Such a strategy proves to be beneficial with respect to the independent processing of images and shows only a small performance loss compared to the joint encoding of different views. The encoding complexity in the previous approach is non-negligible due to the visual information processing based on Slepian-Wolf coding and its associated rate parameter estimation. We therefore discard the Slepian-Wolf encoding and propose a distributed coding solution, where the correlated images are encoded independently using transform-based coding solutions (e.g., SPIHT). The central decoder now builds a correlation model from the compressed images, which is used to jointly decode a pair of images. Experimental results demonstrate that the proposed distributed coding solution improves the rate-distortion performance of the separate coding results for both planar and omnidirectional images. However, this improvement is significant only at medium to high bit rates. We therefore propose a rate allocation scheme that identifies and transmits the necessary visual information from each image to improve the correlation estimation accuracy at low bit rate. Experimental results show that for a given bit budget the proposed encoding scheme permits to compute an accurate correlation estimation comparing to the one obtained with SPIHT, JPEG 2000 or JPEG coding schemes. We show however that the improvement in the correlation estimation comes at the price of penalizing the image reconstruction quality; therefore there exists an interesting trade-off between the accurate correlation estimation and image reconstruction as encoding optimization objectives are different in both cases. Next, we further simplify the encoding complexity by replacing the classical imaging sensors with the simple CS sensors, that directly acquire the compressed images in the form of quantized linear measurements. We now concentrate on the particular problem, where one image is selected as the reference and it is used as a side information for the correlation estimation. We propose a geometry-based model to describe the correlation between the visual information in a pair of images. The joint decoder first captures the most prominent visual features in the reconstructed reference image using geometric functions. Since the images are correlated, these features are likely to be present in the other images too, possibly with geometric transformations. Hence, we propose to estimate the correlation model with a regularized optimization problem that locates these features in the compressed images. The regularization terms enforce smoothness of the transformation field, and consistency between the estimated images and the quantized measurements. Experimental results show that the proposed scheme is able to efficiently estimate the correlation between images for several multi-view and video datasets. The proposed scheme is finally shown to outperform DSC schemes based on unsupervised disparity (or motion) learning, as well as independent coding solutions based on JPEG 2000. We then extend the previous scenario to a symmetric decoding problem, where we are interested to estimate the correlation model directly from the quantized linear measurements without explicitly reconstructing the reference images. We first show that the motion field that represents the main source of correlation between images can be described as a linear operator. We further derive a linear relationship between the correlated measurements in the compressed domain. We then derive a regularized cost function to estimate the correlation model directly in the compressed domain using graph-based optimization algorithms. Experimental results show that the proposed scheme estimates an accurate correlation model among images in both multi-view and video imaging scenarios. We then propose a robust data fidelity term that improves the quality of the correlation estimation when the measurements are quantized. Finally, we show by experiments that the proposed compressed correlation estimation scheme is able to compete the solution of a scheme that estimates a correlation model from the reconstructed images without the complexity of image reconstruction. Finally, we study the benefit of using the correlation information while jointly reconstructing the images from the compressed linear measurements. We consider both the asymmetric and symmetric scenarios described previously. We propose joint reconstruction methodologies based on a constrained optimization problem which is solved using effective proximal splitting methods. The constraints included in our framework enforce the reconstructed images to satisfy both the correlation and the quantized measurements consistency objectives. Experimental results demonstrate that the proposed joint reconstruction scheme improves the quality of the decoded images, when compared to a scheme where the images are handled independently. In this thesis we build efficient distributed scene representation algorithms for the multiple correlated images captured in planar, omnidirectional and CS cameras. The coding rate in our symmetric distributed coding solution stays balanced between the encoders and stays close to the joint encoding solutions. Our novel algorithms lead to effective correlation estimation in different sensing and coding scenarios. In addition, we provide innovative solutions for robust correlation estimation from highly compressed images in simple sensing frameworks. Our CS-based joint reconstruction frameworks effectively exploit the inter-view correlation, that permits to achieve high compression gains compared to state-of-the-art independent and distributed coding solutions.

EPFL2012