Computational methods for live heart imaging with speed-constrained microscopes

Imaging methods to capture the beating and developing heart inside embryonic animal models such as the zebrafish are a key component for the study of fundamental biological processes such as cardiac birth defects or tissue regeneration. However, live heart imaging is subject to many challenges that limit the achievable spatial and temporal resolutions. While many commonly available microscopes offer the necessary optical sectioning capabilities to produce slices or volumes at the required scale, they often lack the necessary speed to seamlessly resolve the beating heart within a still frame, the entire heartbeat, or over a complete developmental sequence. In this thesis, I aim to assemble dynamic image series of the heart by using a wide range of microscopes, including, in particular, scanning microscopes---which are widely available but slow---and microscopes with a slow framerate. First, I considered the problem of reconstructing an image sequence covering one heartbeat from still images arbitrarily triggered throughout multiple cardiac cycles, when the underlying cardiac motion is temporally asymmetrical. I proposed to sort the images such as to maximize the overall similarity between contiguous frames, which I formulated as a TSP. The performance of the method increases with the number of images. I reconstructed image sequences of the heart at a virtual frame-rate of up to 300 frames per second. Second, to resolve the ill-posedness of the sorting problem when identical object poses appear multiple times in one cycle, I proposed to illuminate alternate frames with a temporally-asymmetric pattern. Sorting issues are mitigated at the cost of a custom setup and additional light exposure, without adversely affecting the reconstructed sequence. Third, I considered the problem of imaging the beating heart on conventional scanning confocal microscopes (LSM) that have a frame scanning rate comparable to the heartbeat rate. My approach takes a set of arbitrarily triggered images that may contain scanning aberrations and assembles it into a sequence that covers one heartbeat. Reconstructing scanning-aberrated frames offers a robust alternative for LSM. The quality of recovered cardiac heartbeat dynamics is comparable to that obtained by fast microscopes. Finally, I considered the problem of building smooth time-lapse sequences in the cardiac phase versus development or imaging depth spaces, based on a set of still images captured at weakly-constrained times within the heartbeat and over the course of development or at various depths. This method determines a smooth path in cardiac phase-development or depth space enforcing neighborhood similarity via a within-stage (or within-depth) sorting followed by a cyclical least-squares phase alignment approach that I formulated as a mixed integer linear problem. Constraining the paths of time-lapses by providing key-points while enforcing local similarity successfully stabilizes the the excursion error from which recursive registration approaches suffer. The results suggest that reliably imaging the heart on a wide range of microscopes is possible, provided the heart motions are stereotypical and that minimal sampling constraints are met. The newly gained possibility of using conventional instruments may be particularly relevant for applications with emerging modalities that do not allow wide-field capture.

Reconstruction of image sequences from ungated and scanning-aberrated laser scanning microscopy images of the beating heart

Michael Stefan Daniel Liebling, Olivia Mariani

Fluorescence laser-scanning microscopy is a well-established imaging technique in biology, available in many imaging facilities to investigate structures within live animal embryos such as zebrafish. Laser scanning microscopes (LSM) are limited when used to study dynamic heart morphology or function. Despite their ability to resolve static cardiac structures, the fast motion of the beating heart introduces severe artifacts in the scanned images and gating the acquisitions to the heartbeat is difficult to implement on traditional microscopes. Furthermore, although alternative high-speed imaging instruments exist, they are not widely available (due to cost or hardware complications), putting dynamic cardio-vascular imaging off-limits for many researchers. Here, we propose a method that allows imaging the beating heart on conventional LSMs. Our approach takes a set of images containing scanning aberrations, each triggered at an arbitrary time in the cardiac cycle, and assembles an image sequence that covers a single cardiac heartbeat. The steps are: (i) frame sorting by solving a traveling salesman problem; (ii) heartbeat duration estimation; and (iii) scan-delay compensation via space-time resampling. We characterize the performance of our method on synthetic data under several light intensities and scanning speeds. We further illustrate our method's applicability on experimental images acquired in live zebrafish larvae, and show that the reconstruction quality approaches that of fast, state-of-the-art microscopes. Our technique opens the possibility of using LSMs to carry out studies of cardiac dynamics, without the need for prospective gating or fast microscopes.

2019

Application of CMOS image sensors in optical sectioning and polarization microscopy

Jelena Mitic

In this thesis new imaging approaches for optical microscopy are proposed and studied. They are based on the use of dynamic structured illumination in combination with a demodulation detection concept employing CMOS image detectors. Two particular implementations are suggested: real-time optical sectioning microscopy and whole field quantitative polarization microscopy. Optical sectioning, i.e depth resolved imaging, allows observation of thin sections in volumetric samples. In its classical configuration the wide-field microscope does not allow optical sectioning of the investigated objects. In this thesis the optical sectioning in a wide-field microscope is achieved by illuminating the sample with a continuously moving single spatial frequency pattern, which temporarily modulates the signal obtained on the photodetector. Only the object parts that are in the focus have a good contrast and consequently generate stronger signal on the detector. The optically sectioned images are obtained employing a CMOS image detector, which demodulates the time-dependent component of the image. Two different systems for real-time acquisition of optically sectioned images are proposed. The first one is based on a specially designed smart-pixel-detector-array (SPDA), which allows performing the signal processing by an electronic circuit integrated to each pixel of the sensor. The second system utilizes a commercial CMOS image sensor. Here, the images are treated by a digital signal processor (DSP) integrated into the camera (iMVS-155). Both approaches provide real-time acquisition of optically sectioned images. The proof of principle for the new method is demonstrated by imaging artificial three-dimensional reflective samples. The optical sectioning imaging of biological samples in bright-field mode and in fluorescence mode is also demonstrated. The optical sectioning performances of the studied systems are similar to that of the confocal microscope. However the new method has some advantages over the classical confocal system: e.g. the difficulties with the alignment and the post-processing inherent to the confocal system are avoided. The theoretical framework presented in the thesis describes the image formation in structured illumination for both reflective and fluorescent samples. The influence of longitudinal chromatic aberration and spherical aberration caused by specimen induced index mismatch on depth discrimination property are studied. In extension to the work related to optically sectioned microscopy, the quantitative polarization microscopy imaging method is proposed as a new application for CMOS detectors. The polarization state of the illuminating light is dynamically changed and the demodulation detection approach is employed to extract the retardance and azimuth of the birefringent sample in real-time; for a whole field of view. Examples of polarization sensitive measurements for different samples are presented. The theoretical analysis is performed using the Jones formalism. Finally, the potentials and limitations of the CMOS detectors for applications in optical microscopy are discussed. Thanks to constant advancements in CMOS technology, the acquisition speed, resolution and sensitivity of new CMOS detectors generations have been improved. This will allow adapting the methods proposed in this thesis for investigations of fast dynamic process where high temporal and spatial resolution is required.

EPFL2004

Optical microscopy for imaging through scattering media

Xin Yang

Optical microscopy has been widely used in the life sciences and biological research for several decades. It possesses many advantages over x-ray radiography, computed tomography, ultrasound imaging and magnetic resonance imaging, in terms of being low cost, high resolution, harmless and capable of multiple contrast mechanism, etc. However, the weakness of optical microscopy is the inability to perform deep tissue imaging. Owing to the short wavelength of optical waves, it experiences much more scattering than longer waves in inhomogeneous materials such as biological tissues. Scattering scrambles the wavefront and limits the penetration depth of optical microscopy to only hundreds of microns. We explore in this thesis, novel optical approaches to image through scattering media. There are two main categories of the state of art techniques: imaging with ballistic photons, which requires separation of ballistic photons from the sneak or multiple scattered ones; and imaging with scattered photons, which relies on manipulation of the scattered photons according to the specific scattering properties. We primarily focus on the second category in this thesis. Taking advantage of the reversibility of light, phase conjugation has a long history in aberration compensation. A deterministic wave is incident on a scattering medium and results in a random scattered field. If we record the full information of the scattered field (amplitude and phase) and generate a phase conjugated replica of it, the phase conjugated wave can follow the same path and reconstruct the original deterministic wave after passing back through the scattering medium. In this work, phase conjugation is achieved using digital method, called digital phase conjugation. The scattered field is recorded through digital holography, and a spatial light modulator is applied to construct the phase conjugated wave. Second harmonic radiation imaging probes are used as the original light source, rendering the generation of a focus behind the scattering medium. The memory effect, which indicates the correlation between speckles with different incident angles, is exploited to move the focus around and obtain three-dimensional scanning microscopy behind scattering medium. Speckle scanning microscopy is the second method demonstrated in this thesis for imaging through turbid media. In contrast to digital phase conjugation, which requires access from both sides of the scatterers to measure the scattered field, in speckle scanning microscopy, excitation and collection can be achieved from the same side. The specific scattering events are not taken into account, instead, the statistic property of the scatterers plays a major role. This technique makes full use of two facts: 1. The statistically averaged autocorrelation of speckle fields approaches a delta function; 2. When a speckle pattern scans across a fluorescence object, the collected fluorescence intensity is proportional to the convolution of the speckle field and the object. Linking these two facts with delicate calculation, we can eventually minimize the influence of the speckle, and reconstruct two-dimensional images of objects behind turbid medium. Encouraged by the success of two-dimensional speckle scanning microscopy, we investigate more advanced possibilities by simulation, including three-dimensional speckle scanning and reference based speckle scanning techniques. [...]