Three-dimensional tomography of red blood cells using deep learning

Ahmed Ayoub, Joowon Lim, Demetri Psaltis
2020
Journal paper

Abstract

We accurately reconstruct three-dimensional (3-D) refractive index (RI) distributions from highly ill-posed two-dimensional (2-D) measurements using a deep neural network (DNN). Strong distortions are introduced on reconstructions obtained by the Wolf transform inversion method due to the ill-posed measurements acquired from the limited numerical apertures (NAs) of the optical system. Despite the recent success of DNNs in solving ill-posed inverse problems, the application to 3-D optical imaging is particularly challenging due to the lack of the ground truth. We overcome this limitation by generating digital phantoms that serve as samples for the discrete dipole approximation (DDA) to generate multiple 2-D projection maps for a limited range of illumination angles. The presented samples are red blood cells (RBCs), which are highly affected by the ill-posed problems due to their morphology. The trained network using synthetic measurements from the digital phantoms successfully eliminates the introduced distortions. Most importantly, we obtain high fidelity reconstructions from experimentally recorded projections of real RBC sample using the network that was trained on digitally generated RBC phantoms. Finally, we confirm the reconstruction accuracy using the DDA to calculate the 2-D projections of the 3-D reconstructions and compare them to the experimentally recorded projections.

Official source

https://infoscience.epfl.ch/record/286015?ln=en

About this result

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Related concepts

Related publications

Learning approaches to high-fidelity optical diffraction tomography

Joowon Lim

Optical diffraction tomography (ODT) provides us 3D refractive index (RI) distributions of transparent samples. Since RI values differ across different materials, they serve as endogenous contrasts. It, therefore, enables us to image without pre-processing of labeling which can disturb samples during measurement. It has been utilized in various applications to study hematology, morphological parameters, biochemical information, and so on. The fundamental principle of ODT reconstruction is to recover the 3D information from multiple 2D measurements. While we require 2D measurements acquired by fully scanning a sample, there exist missing measurements that we are not able to access due to the limited numerical apertures (NAs) in the optical system. This is called the missing cone problem since the parts which are not covered by the NAs form cone shapes. The missing cone problem degrades the final reconstruction by underestimating RI values and more severely elongating images along the optical axis. Another challenge in ODT reconstruction is to model the nonlinear relationship between a sample and the measurements. The first order of scattering is commonly considered while neglecting the other higher orders to linearize the relationship, however, this results in degradation of the final reconstruction as the higher orders of scattering become more pronounced. In this thesis, we aim at solving the challenges in ODT reconstruction to provide more accurate quantitative information, namely, RI distributions. The first approach is based on model-based iterative reconstruction schemes. We choose the beam propagation method (BPM) for the forward model in order to capture the high orders of scattering. Due to the similarity of the multi-layer structure of the BPM with that of neural networks used in deep learning, we call this scheme learning tomography (LT). We rigorously investigate the performance of LT over the conventional linear model-based reconstruction scheme. Furthermore, by applying a more advanced BPM for the forward model, we even improve the LT and demonstrate the dramatically improved performance by both simulations and experiments. The second approach is based on statistically learning artifacts present in final reconstructions using a deep neural network (DNN) from a large dataset. Unlike the previous approaches which require iterations, the DNN approach instantly reconstructs RI distributions. We demonstrate the use of DNN using red blood cells which are highly distorted by the missing cone problem. In order to overcome the lack of ground truth in 3D ODT reconstruction, we digitally generate a synthetic dataset. The reconstruction results from the network present highly accurate results for the synthetic test set. Most importantly, we obtain high-fidelity reconstructions of experimental data using the network trained only on the synthetic data. Unlike other imaging modalities, ODT provides 3D quantitative information without labeling. To fully benefit from the capacity of quantitative imaging, it is critical to solve the existing challenges in ODT reconstruction to produce high-fidelity reconstructions. In this contribution, we aim to resolve the major challenges in ODT reconstruction using various learning approaches, and we believe that it can further improve ODT as a powerful tool for various applications.

EPFL2020

3D Reconstruction of Optical Diffraction Tomography Based on a Neural Network Model

Morteza Hasani Shoreh

Optical tomography has been widely investigated for biomedical imaging applications. In recent years, it has been combined with digital holography and has been employed to produce high quality images of phase objects such as cells. In this Thesis, we look into some of the newest optical Diffraction Tomography (DT) based techniques to solve Three-Dimensional (3D) reconstruction problems and discuss and compare some of the leading ideas and papers. Then we propose a neural-network-based algorithm to solve this problem and apply it on both synthetic and biological samples. Conventional phase tomography with coherent light and off axis recording is performed. The Beam Propagation Method (BPM) is used to model scattering and each x-y plane is modeled by a layer of neurons in the BPM. The network's output (simulated data) is compared to the experimental measurements and the error is used for correcting the weights of the neurons (the refractive indices of the nodes) using standard error back-propagation techniques. The proposed algorithm is detailed and investigated. Then, we look into resolution-conserving regularization and discuss a method for selecting regularizing parameters. In addition, the local minima and phase unwrapping problems are discussed and ways of avoiding them are investigated. It is shown that the proposed learning tomography (LT) achieves better performance than other techniques such as, DT especially when insufficient number or incomplete set of measurements is available. We also explore the role of regularization in obtaining higher fidelity images without losing resolution. It is experimentally shown that due to overcoming multiple scattering, the LT reconstruction greatly outperforms the DT when the sample contains two or more layers of cells or beads. Then, reconstruction using intensity measurements is investigated. 3D reconstruction of a live cell during apoptosis is presented in a time-lapse format. At the end, we present a final comparison with leading papers and commercially available systems. It is shown that -compared to other existing algorithms- the results of the proposed method have better quality. In particular, parasitic granular structures and the missing cone artifact are improved. Overall, the perspectives of our approach are pretty rich for high-resolution tomographic imaging in a range of practical applications.

EPFL2018

In this thesis, we propose new algorithms to solve inverse problems in the context of biomedical images. Due to ill-posedness, solving these problems require some prior knowledge of the statistics of the underlying images. The traditional algorithms, in the field, assume prior knowledge related to smoothness or sparsity of these images. Recently, they have been outperformed by the second generation algorithms which harness the power of neural networks to learn required statistics from training data. Even more recently, last generation deep-learning-based methods have emerged which require neither training nor training data. This thesis devises algorithms which progress through these generations. It extends these generations to novel formulations and applications while bringing more robustness. In parallel, it also progresses in terms of complexity, from proposing algorithms for problems with 1D data and an exact known forward model to the ones with 4D data and an unknown parametric forward model. We introduce five main contributions. The last three of them propose deep-learning-based latest-generation algorithms that require no prior training. 1) We develop algorithms to solve the continuous-domain formulation of inverse problems with both classical Tikhonov and total-variation regularizations. We formalize the problems, characterize the solution set, and devise numerical approaches to find the solutions. 2) We propose an algorithm that improves upon end-to-end neural-network-based second generation algorithms. In our method, a neural network is first trained as a projector on a training set, and is then plugged in as a projector inside the projected gradient descent (PGD). Since the problem is nonconvex, we relax the PGD to ensure convergence to a local minimum under some constraints. This method outperforms all the previous generation algorithms for Computed Tomography (CT). 3) We develop a novel time-dependent deep-image-prior algorithm for modalities that involve a temporal sequence of images. We parameterize them as the output of an untrained neural network fed with a sequence of latent variables. To impose temporal directionality, the latent variables are assumed to lie on a 1D manifold. The network is then tuned to minimize the data fidelity. We obtain state-of-the-art results in dynamic magnetic resonance imaging (MRI) and even recover intra-frame images. 4) We propose a novel reconstruction paradigm for cryo-electron-microscopy (CryoEM) called CryoGAN. Motivated by generative adversarial networks (GANs), we reconstruct a biomolecule's 3D structure such that its CryoEM measurements resemble the acquired data in a distributional sense. The algorithm is pose-or-likelihood-estimation-free, needs no ab initio, and is proven to have a theoretical guarantee of recovery of the true structure. 5) We extend CryoGAN to reconstruct continuously varying conformations of a structure from heterogeneous data. We parameterize the conformations as the output of a neural network fed with latent variables on a low-dimensional manifold. The method is shown to recover continuous protein conformations and their energy landscape.

EPFL2020

3D Reconstruction of Optical Diffraction Tomography Based on a Neural Network Model

Morteza Hasani Shoreh

EPFL2018

EPFL2020

Learning approaches to high-fidelity optical diffraction tomography

Joowon Lim

EPFL2020